Apparatus selectively presenting distortion corrected image data

ABSTRACT

Relative to an image processing apparatus that processes image data containing a distortion of an image pickup optical section, which is obtained by picking up an optical image from a subject through the image pickup optical section giving the distortion, an image processing apparatus, an image processing method, a program therefor, a recording medium in which the program is recorded, an image pickup apparatus, which are easily used by the user conveniently, are provided. A data output section ( 13   d ) outputs the image data of a display image that corresponds to a display mode by using a subject image whose selected region which indicates a part of a field of view represented by the image data is made identifiable and a distortion-corrected image of the selected region. A display mode setting section ( 13   c ) performs setting of the display mode. In accordance with switching of the selected region, a control section ( 13   e ) changes a first display mode in which the subject image whose selected region is made identifiable is not used in the display image to a second display mode in which the subject image whose selected region is made identifiable is used in the display image.

This application is a 371 U.S. National Stage filing ofPCT/JP2006/322499, filed Nov. 10, 2006, which claims priority toJapanese Patent Application Number JP2005-327749 filed Nov. 11, 2005 andJapanese Patent Application Number JP2006-176915 filed Jun. 27, 2006,all of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to an image processing apparatus, an imageprocessing method, a program therefor, a recording medium in which theprogram is recorded, and an image pickup apparatus, which performsprocessing on a picked-up wide-field image.

BACKGROUND ART

Conventionally, a technology has been proposed by which a user specifiesa necessary region in an image picked up by using, for example, afish-eye lens, a distortion aberration of the fish-eye lens image dataof the specified region is corrected, and a corrected image is displayedon a monitor. The picked-up image may be a moving image as well as astill image (see, for example, Japanese Patent Application PublicationNo. 2000-324386 (see FIG. 1 and in paragraph [0009] in thedescription)).

With an apparatus described in this Japanese Patent ApplicationPublication No. 2000-324386, it is expected that a higher degree ofconvenience is given if a user devices a better method for specifying aregion in an image picked-up with a camera. Such a system that uses thecamera is expected to be used in wide applications in the future and somay have an important task to provide the user with an interface easierto use.

DISCLOSURE OF INVENTION

An image processing apparatus according to the present invention is animage processing apparatus that processes image data containing adistortion of an image pickup optical section, which is obtained bypicking up an optical image from a subject through the image pickupoptical section giving the distortion, the image processing apparatuscomprising a data output section that outputs the image data of adisplay image that corresponds to a display mode by using a subjectimage whose selected region which indicates a part of a field of viewrepresented by the image data is made identifiable and adistortion-corrected image of the selected region, a display modesetting section that sets the display mode, and a control section that,in accordance with switching of the selected region, changes a firstdisplay mode in which the subject image whose selected region is madeidentifiable is not used in the display image to a second display modein which the subject image whose selected region is made identifiable isused in the display image. An “image” in the present invention mainlymeans a moving image but, of course, includes a still image.

The present invention is provided with an input section that, inaccordance with the switching of the selected region, performsswitchover instruction of the selected region, for example, and if theselected region is switched by this switchover instruction, the displaymode is changed for a predetermined period of time. Further, a firstregion selection mode in which a selected region is switched inorthogonal coordinates based on the switchover instruction and a secondregion selection mode in which the selected region is switched in polarcoordinates based on the switchover instruction are provided, so thatwhen the region selection mode is switched, the display mode is changedfor a predetermined period of time. An orthogonal coordinate mode is aparticularly effective mode to, for example, a field of view ordinarilywatched by a human being, that is, in the case of picking up an imagesuch as a landscape having up and down, right and left. Further, a polarcoordinate mode is a particularly effective mode in the case of pickingup an image having such a field of view as to, for example, watch upwardor downward from a horizontal plane. That is, by changing the regionselection mode, it is possible to perform a highly intuitive operationby the human being. Thus, by changing the display mode for thepredetermined period of time if a selected region or the regionselection mode is switched, it is possible to provide display differentfrom ordinary display for a predetermined period of time when theselected region is switched, thereby enabling image display thateffectively utilizes a display section to be realized. Further, adirection detection sensor that detects an image pickup direction isprovided so that when it is determined on the basis of a sensor signalfrom the direction detection sensor that a selected region is changed,even if the display mode is changed for the predetermined period oftime, it is also possible to provide an image display that effectivelyutilizes the display section.

Herein, the control section changes for the predetermined period of timethe display mode in which the subject image whose selected region ismade identifiable is not used in the display image to the display modein which the subject image whose selected region is made identifiable isused in the display image. In this case, if the selected region ischanged or the region selection mode is switched, the subject imagewhose selected region is made identifiable is displayed for apredetermined period of time, so that the setting state of the selectedregion can be confirmed. Further, after the predetermined period of timehas lapsed, it is switched to the display mode in which the subjectimage whose selected region is made identifiable is not used in thedisplay image, so that even if a blind region occurs due to the displayof the subject image whose selected region is made identifiable in, forexample, a distortion-corrected image of the selected region, after thepredetermined period of time has lapsed, the image of this region can beconfirmed. It should be noted that the predetermined period of time maybe, but not limited to, three seconds, five seconds, etc., for example.

An image processing method according to the present invention is animage processing method that processes image data containing adistortion of an image pickup optical section, which is obtained bypicking up an optical image from a subject through the image pickupoptical section giving the distortion, the image processing methodcomprising a data output step of outputting the image data of a displayimage that corresponds to a display mode by using a subject image whoseselected region which indicates a part of a field of view represented bythe image data is made identifiable and a distortion-corrected image ofthe selected region, a display mode setting step of setting the displaymode, and a display mode changing step of, in accordance with switchingof the selected region, changing a first display mode in which thesubject image whose selected region is made identifiable is not used inthe display image to a second display mode in which the subject imagewhose selected region is made identifiable is used in the display image.The present invention can also be applied to an invention of a programand inventions of a recording medium in which this program is recordedand an image pickup apparatus.

As described above, by the present invention, through image processingthat can reproduce an image obtained by selecting a region in an imagepicked up using an image pickup optical section, it is possible torealize an image processing apparatus or an image processing method etc.that can be easily used by the user conveniently.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a constitution of an image processingsystem according to one embodiment of the present invention.

FIG. 2 is an explanatory illustration of a relationship between asubject image formed on an image pickup element and a selected region.

FIG. 3 is a block diagram showing a functional constitution of an imageprocessing section.

FIG. 4 are explanatory diagrams of a display mode and a region selectionmode.

FIG. 5 is a block diagram exemplifying a specific constitution of theimage processing section.

FIG. 6 is an illustration showing an entire image.

FIG. 7 is an illustration showing one example of an image which isdisplayed on a display section.

FIG. 8 is a flowchart showing operations of a processing controlsection.

FIG. 9 is an illustration showing a field of view as a sphere of athree-dimensional space.

FIG. 10 are illustrations for showing the respective fields of view(image pickup ranges) in picking up by an image pickup section.

FIG. 11 are explanatory diagrams of image height characteristics of alens.

FIG. 12 are explanatory diagrams of principles of distortion correctionprocessing.

FIG. 13 is an explanatory illustration of operations in an orthogonalcoordinate mode out of region selection modes.

FIG. 14 is an explanatory illustration of operations in a polarcoordinate mode out of the region selection modes.

FIG. 15 show one example of a GUI which is displayed when a userinstructs changing a selected region through an input section.

FIG. 16 are explanatory illustrations of a case where it is instructedto switch the selected region when the orthogonal coordinate mode isselected.

FIG. 17 show display images which are displayed on a display section 14when the display modes are sequentially switched when the orthogonalcoordinate mode is selected.

FIG. 18 are illustrations showing display images which are displayed onthe display section when the polar coordinate mode is selected.

FIG. 19 is an illustration showing a display image after a switchoverinstruction is issued when the polar coordinate mode is selected.

FIG. 20 is an illustration showing a display image displayed on thedisplay section when the polar coordinate mode is selected, andexplaining a case where a display in a divided display mode is turnedupside down.

FIG. 21 are illustrations showing transition of the display modeswitchover in a case where selection of the divided display mode isenabled.

FIG. 22 are explanatory illustrations of scale up/down processing for animage displayed in a selected region.

FIG. 23 are illustrations showing a state before a rotation instructionis issued.

FIG. 24 are explanatory illustrations of processing to rotate an imagein a selected region.

FIG. 25 are explanatory illustrations of another piece of processing torotate an image in a selected region.

FIG. 26 is a block diagram showing a constitution of an image processingsystem according to another embodiment of the present invention.

FIG. 27 is a flowchart showing one example of processing which isperformed by the image processing system shown in FIG. 26.

FIG. 28 is a flowchart showing another example of the processing whichis performed by the image processing system shown in FIG. 26.

FIG. 29 is a flowchart showing processing to store position informationor trajectory information.

FIG. 30 is an explanatory diagram of a trajectory of a predeterminedrange on a displayed image along the flow shown in FIG. 29.

FIG. 31 is a block diagram showing a constitution of an image processingsystem according to further embodiment of the present invention.

FIG. 32 is a block diagram showing a constitution of an image processingsystem according to still further embodiment of the present invention.

FIG. 33 are conceptual illustrations of a manner to switch a regionselection mode MS in accordance with a direction in which an imagepickup section 11 is placed in the image processing system of FIG. 32.

FIG. 34 are explanatory illustrations of a method of setting thresholdvalues for switching states S₀, S₁, and S₂ of FIG. 33.

FIG. 35 is a flowchart showing a flow of operations in a case where theimage processing system 40 of FIG. 32 switches states S₀, S₁, and S₂from each other;

FIG. 36 are explanatory diagrams of a coordinate calculation method in acase where the image processing system of FIG. 32 sets the orthogonalcoordinate mode and the polar coordinate mode, respectively.

FIG. 37 is an explanatory diagram of the coordinate calculation methodin a case where the image processing system of FIG. 32 sets theorthogonal coordinate mode and the polar coordinate mode, respectively.

FIG. 38 are explanatory diagrams of the coordinate calculation method ina case where the image processing system of FIG. 32 sets the orthogonalcoordinate mode and the polar coordinate mode, respectively.

FIG. 39 are explanatory diagrams of the coordinate calculation method ina case where the image processing system of FIG. 32 sets the orthogonalcoordinate mode and the polar coordinate mode, respectively.

FIG. 40 is a conceptual illustration of a method for switching thedisplay mode in accordance with contact.

FIG. 41 is a diagram showing a case where the image pickup opticalsection has obtained a field of view of 270 degrees.

FIG. 42 is a diagram showing a state where an image pickup direction isset to 45 degrees upward with respect to the horizontal direction.

FIG. 43 are illustrations showing an example where the image pickupsection is placed.

FIG. 44 are explanatory illustrations of automatic switchover of theregion selection mode.

FIG. 45 are illustrations of a GUI display and an image region movementdirection in a case where the region selection mode is switchedautomatically.

FIG. 46 is an explanatory illustration of switchover operation among theregion selection modes including a combined mode.

FIG. 47 is a flowchart showing switchover operation among the regionselection modes including the combined mode.

FIG. 48 is a flowchart showing operations when a direction button isoperated.

FIG. 49 are illustrations showing a manner to change the display mode MHin response to switchover of the selected region in additionalembodiment of the present invention.

FIG. 50 are illustrations showing a manner to change the display mode MHin response to switchover of the region selection mode MS in furtheradditional embodiment of the present invention.

FIG. 51 is a flowchart showing a flow of operations of the imageprocessing system in a case of performing display mode switchoverprocessing.

FIG. 52 is a flowchart showing another flow of the operations of theimage processing system in the case of performing display modeswitchover processing.

FIG. 53 are illustrations showing another state of the display modeswitchover processing.

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described below withreference to the drawings.

FIG. 1 is a block diagram showing the constitution of an imageprocessing system according to one embodiment of the present invention.

The image processing system 10 is provided with an image pickup section11, an input section 12, an image processing section 13, and a displaysection 14. Further, the image pickup section 11 is constituted by usingan image pickup optical section 111 and an image pickup element 112.

The image pickup optical section 111 is used to focus a subject image onan imaging area of the image pickup element 112. In this case, as theimage pickup optical section 111, for example, a wide-angle lens is usedto focus a wide-field subject image on the imaging area of the imagepickup element 112. The wide-angle lens has a field angle of, but notlimited to, at least about 45 degrees. On the other hand, the imagepickup optical section 111 may be constituted by using a fish-eye lens,a panoramic annular lens (PAL), which is a kind of annular lens, etc.Further, in place of using the wide-angle lens, a tube-shaped,bowl-shaped, or cone-shaped mirror may be used so that a wide-fieldsubject image may be focused on the imaging area of the image pickupelement 112 through reflection by this mirror. Further, pluralities oflenses and mirrors may be combined to spread the field of view further.For example, by two fish-eye lenses having a field angle of about 180degrees, it is possible to obtain a subject image having an all-roundview (spherical space (360 degrees)).

As the image element 112, for example, a charge coupled device (CCD), acomplementary metal-oxide semiconductor (CMOS) sensor or the like isemployed which converts light into an electrical signal. This imagepickup element 112 generates image data DVa based on a subject image andsupplies it to the image processing section 13.

FIG. 2 shows a relationship between a subject image formed on the imagepickup element 112 and a selected region in a case where a fish-eye lensis employed as the image pickup optical section 111. If the image pickupoptical section 111 has a field angle of, for example, about 180 degreesand its field of view can be represented by a hemispherical sphere 51 inFIG. 2, a subject image (hereinafter referred to as “wide-field image”)Gc, which is formed on the image pickup element 112, will become animage containing distortion due to the image pickup optical section 111,for example, a circular image. Accordingly, when an image is displayedbased on the image data DVa obtained by the image pickup element 112,the displayed image becomes one having distortion due to the imagepickup optical section 111. In this case, if a selected region isprovided which indicates a partial region of a field of view representedby the image data DVa, this selected region corresponds to, for example,a selected region 71 within the field angle of the image pickup opticalsection 111 and also corresponds to an image region ARs in thewide-field image Gc. Therefore, the image processing section 13 canperform distortion correction processing on the image of the imageregion ARs for correcting the distortion due to the image pickup opticalsection 111, to display a distortion-free image of the selected region.Accordingly, by setting such a selected region as to include a desiredsubject within, for example, a field angle of the image pickup opticalsection 111, it is possible to display a distortion-free image of thedesired subject. Furthermore, by switching the position of a selectedregion to a new position or changing the size or shape of this region,the position or the size or shape of the image region ARs correspondingto the selected region will also be changed. It is thus possible todisplay the image of an arbitrary position or region within the fieldangle of the image pickup optical section 111 in a condition where thedistortion due to the image pickup optical section 111 is corrected.

Herein, the selected region can be set by specifying an angle range etc.that indicates a position and a range of the selected region 71 in thefield angle of the image pickup optical section 111. Further, theselected region can be set also by specifying a position, a range, etc.of the image region ARs which is set on the wide-field image Gc becausethe image region Gc corresponds to the selected region as describedabove.

It should be noted that in a case where a fish-eye lens is used, thesmaller the selected region is and the closer it is located to thecenter of a field of view, the more the range of the image region ARsresembles the selected region in shape. Conversely, the larger theselected region is and the closer it is located to an edge of the fieldof view, the range of the image region ARs is distorted with respect tothe selected region in shape.

The input section 12 is used to switch the position of a selectedregion, change the region size and the region shape of the selectedregion, and set the operation mode and the image display mode inswitchover of the selected region, in response to the operations of auser. The input section 12 may be any device as far as it can beoperated by the user. It may be, for example, a mouse, a keyboard, aswitch device, a touch sensor, a game machine controller, a stick-likemanipulator that can be gripped by the user, etc. The input section 12generates input information PS that corresponds to the user operationsand supplies it to the image processing section 13.

The image processing section 13 performs distortion correctionprocessing by using the image data DVa supplied from the image pickupsection 11, to generate an image of the selected region in whichdistortion due to the image pickup optical section 111 is corrected.Further, the image processing section 13 sets a display mode for theimage to be displayed on the display section 14, generates image dataDVd of the display image corresponding to the thus set display mode, andsupplies it to the display section 14. It should be noted that the imageprocessing section 13 uses as the display image the wide-field image,the image of the selected region in which the distortion are corrected,and the like. Further, the image processing section 13 sets a regionselection mode, which is an operation mode activated when switching theposition of a selected region, and switches the selected region in thethus set region selection mode based on the input information PS fromthe input section 12. Also, the image processing section 13 performsprocessing to set it to a display mode or region selection mode, whichis specified beforehand at a time of an initial operation start,processing to set it to a display mode or region selection mode at atime of an operation completion and to start the operation, and thelike.

The display section 14 is constituted of a liquid crystal displayelement, an organic EL element or the like, to display an image based onthe image data DVd supplied from the image processing section 13.

In the image processing system 10 according to the present embodiment,the image pickup section 11, the input section 12, the image processingsection 13, and the display section 14 may be integrated with each otheror separated from each other. Further, only some of them may beintegrated with each other. For example, if the input section 12 and thedisplay section 13 are integrated with each other, it is easily possibleto operate the input section 12 with confirming a display on the displaysection 14. Furthermore, in the image pickup section 11, the imagepickup optical section 111 and the image pickup element 112 may beintegrated with each other or separated from each other.

FIG. 3 is a block diagram showing a functional constitution of the imageprocessing section 13. The image processing section 13 has a distortioncorrection section 13 a, a region selection mode setting section 13 b, adisplay mode setting section 13 c, a data output section 13 d, and acontrol section 13 e.

The distortion correction section 13 a performs distortion correction tocorrect distortion through the image pickup optical section 111 by usingimage data that corresponds to a selected region in the image data DVa,thereby generating distortion-corrected data.

The region selection mode setting section 13 b sets a region selectionmode, which is an operation mode when setting or switching the selectedregion. This region selection mode setting section 13 b is providedwith, as the region selection mode MS, an orthogonal coordinate mode MS1serving as a first region selection mode and a polar coordinate mode MS2serving as a second region selection mode, as shown in FIG. 4, forexample, and then selectively sets either one of these region selectionmodes. Those region selection modes will be respectively describedlater.

The display mode setting section 13 c sets a display mode when adistortion-corrected image etc. is displayed on the display section 14.As shown in FIG. 4, for example, this display mode setting section 13 cis provided with, as the display mode MH, an entire image display modeMH1, a selected image display mode MH2, a both display mode MH3, and adivided display mode MH4 and then sets any one of these display modes.Those display modes will be respectively described later.

The data output section 13 d outputs the image data of a display imagethat corresponds to the thus set display mode. Herein, it outputsdistortion-corrected data if an image in a selected region whosedistortion due to the image pickup optical section 111 is corrected, isdisplayed as the display image. Further, if a wide-field image isdisplayed as the display image, it outputs image data supplied from theimage pickup section 11. Furthermore, if a wide-field image and an imagein a selected region whose distortion due to the image pickup opticalsection 111 is corrected are displayed, it generates new image data byusing the distortion-corrected data and the image data supplied from theimage pickup section 11 and outputs it.

The control section 13 e sets or switches the selected region inaccordance with the region selection mode MS which is set by the regionselection mode setting section 13 b.

FIG. 5 is a block diagram exemplifying a specific constitution of theimage processing section 13. The image data DVa is supplied to an imageextraction processing section 131 and a distortion correction processingsection 133.

The image extraction processing section 131 extracts image data DVc onthe wide-field image (subject image) Gc from the image data DVa andsupplies it to a selected region highlight display processing section132. Herein, as shown in FIG. 2, the wide-field image Gc indicates apartial region on the sensor surface of the image pickup element 112 andis determined by the image pickup optical section 111. Accordingly, ifthe region of the wide-field image Gc is fixed on the sensor surface,predetermined image data is extracted from the image data DVa and pixeldata of a region of the wide-field image Gc is extracted. Further, ifthe image pickup optical section 111 can be replaced and the region ofthe wide-field image Gc changes on the sensor surface or if the regionof the wide-field image Gc changes on the sensor surface because theoptical characteristics of the image pickup optical section 111 can bechanged, it identifies the region of the wide-field image Gc on thesensor surface beforehand and then extracts the image data of thisidentified region of the wide-field image Gc. The region of thewide-field image Gc is identified by, for example, filling the entiretyof a view of field by the image pickup optical section 111 with a whitesubject, picking it up, and detecting pixel positions at which the imagedata DVa takes on a white level. It is thus possible to identify theregion of the wide-field image Gc easily.

The selected region highlight display processing section 132 performsprocessing such that the user can easily distinguish between a selectedregion indicated by selected region setting information JA supplied froma later-described processing control section 135 and the correspondingimage region ARs, in the wide-field image Gc, based on this selectedregion setting information JA. For example, the selected regionhighlight display processing section 132 conducts display control sothat the image region ARs can be identified by providing boundarydisplay that indicates a boundary between the image region ARs and theregion other than this image region ARs or by changing luminance orcolor of the region other than the image region ARs. The image of thisimage region ARs is highlighted so as to be identifiable and hereinafterreferred to as a highlighted image Gs. By performing such imageprocessing, image data DVcp of an image (hereinafter referred to as“entire image Gcp”) whose image region ARs is made identifiable as thehighlighted image Gs in the wide-field image Gc is generated andsupplied to an image output processing section 134.

The distortion correction processing section 133, which corresponds tothe distortion correction section 13 a shown in FIG. 3, correctsdistortion due to the image pickup optical section 111 by using theimage data DVa to generate corrected image data DVsc in which thedistortion of the selected region indicated by the selected regionsetting information JA supplied from the processing control section 135is corrected and supplies it to the image output processing section 134.

The image output processing section 134, which corresponds to the dataoutput section 13 d shown in FIG. 3, generates the image data DVd of adisplay image by using the image data DVcp and/or the corrected imagedata DVsc based on display control information JH from the processingcontrol section 135.

The processing control section 135 corresponds to the region selectionmode setting section 13 b, the display mode setting section 13 c, andthe control section 13 e. The processing control section 135 sets aregion selection mode, sets or switches a selected region based on theinput information PS from the input section 12 in accordance with thethus set region selection mode, generates selected region settinginformation JA which indicates a set selected region or a newly setselected region, and supplies it to the selected region highlightdisplay processing section 132 and the distortion correction processingsection 133. Further, the processing control section 135 sets a displaymode, generates display control information JH in accordance with thethus set display mode, and supplies it to the image output processingsection 134. The processing control section 135 further performsprocessing to contain a menu display in a display image based on thedisplay control information JH or the like.

The image processing section 13 is constituted of, for example, hardwaresuch as a central processing unit (CPU), a random access memory (RAM),and a read only memory (ROM), software stored in the ROM etc., firmwareor the like. Further, the image processing section 13 may be constitutedof a field programmable gate array (FPGA) or a digital signal processor(DSP) and the like, and also, may be provided with a video encoder, asound encoder, an interface for acquiring the above-described inputinformation PS, an interface for outputting the image data DVd to theabove-described display section 14 or the like. Further, both of theFPGA and the DSP may be used in such a manner that they would performtheir own tasks separately.

The input information PS to be supplied from the input section 12 to theimage processing section 13 is information indicating theabove-described setting of the display mode MH and the region selectionmode MS, instructions to switch the selected region, etc. Further, theinformation that indicates the instructions to switch the selectedregion may include information to move a selected region inpredetermined units in a predetermined direction, information thatindicates the changed position of the selected region, etc. The inputinformation PS may further include information to change the region sizeof a selected region, information to rotate the selected region,information to set a region shape of the selected region, etc.

A selected region is not limited to an embodiment in which switchoveretc., are performed in accordance with a user operation. For example,such an embodiment may be thought of that a selected region would be setto a position specified beforehand as described above. In this case,information about this selected region may be stored beforehand in anROM, external storage device, not shown, or the like. Further, in a casewhere a specific region in a wide-field image Gc is automaticallyrecognized, such an embodiment may also be considered that thisautomatically recognized specific region would be processed as an imageregion ARs that corresponds to a selected region. For example, in a casewhere a moving subject is automatically recognized, such an embodimentmay be thought of that a region including this moving subject would beprocessed automatically as an image region ARs. Alternatively, such anembodiment may be possible that an image region detected in a wide-fieldimage Gc by a variety of sensors, not shown, would be processed as animage region ARs that corresponds to a selected region. In this case,the sensor may be, for example, a temperature sensor, a sound sensor, apressure sensor, an optical sensor, a humidity sensor, a vibrationsensor, a gas sensor, or any other one of the various sensors. A sensorsignal generated by such a sensor is supplied to the processing controlsection 135, which then utilizes this sensor signal to switch a selectedregion or control the respective sections. For example, if a defectivetemperature or sound is detected, by automatically switching a selectedregion corresponding to a direction in which the defective temperatureor sound is detected, a picked up image of the direction in which thedefective temperature or sound is detected can be displayed on a screenof the display section 14 in a condition where its distortion iscorrected. Further, by switching the display mode, the size or shape ofa region, etc. in response to detection of a defect, it is possible todisplay the image in such a manner that the user can easily confirm thedefect.

It should be noted that the constitution shown in FIG. 5 is justillustrative and the image processing section 13 is not limited to thatof the constitution shown in FIG. 5 as far as it has such a mechanism asshown in FIG. 3. For example, the image extraction processing section131 need not be provided if image data DVa supplied from the imagepickup section 11 indicates only a wide-field image Gc. Further, theselected region highlight display processing section 132 may be placedat an output side instead of an input side of the image outputprocessing section 134. In this case, if a wide-field image is includedin an image based on the image data DVa, an image region thatcorresponds to a selected region in the wide-field image will beprocessed in such a manner that it can be easily identified by the user.

FIG. 6 shows an entire image. If distortion correction is not performedon the image data DVa by the image processing section 13, a wide-fieldimage Gc becomes an image containing a distortion generated through theimage pickup optical section 111. Herein, if a selected region is set bythe processing control section 135, the selected region highlightdisplay processing section 132 performs image processing so that aselected region can be easily identified as described above. That is,boundary display (for example, frame display) may be provided toindicate a boundary between the image region ARs corresponding to aselected region and the region other than the image region ARs in adisplayed wide-field image Gc or the luminance or color of the regionother than the image region ARs is changed to provide a highlightedimage Gs in which the image region ARs is highlighted. It should benoted that if a selected region is set all over a field of view, theimage region ARs corresponding to the selected region indicates theentirety of the wide-field image Gc.

Further, the processing control section 135, if having set the entireimage display mode MH1 by controlling the operations of the image outputprocessing section 134 based on the display control information JH,controls the image output processing section 134 to generate image dataDVd by which only an entire image Gcp is to be displayed in which animage region ARs can be identified as a highlighted image Gs in awide-field image Gc such as shown in FIG. 6. Further, if having set theselected image display mode MH2, the processing control section 135controls the image output processing section 134 to generate image dataDVd by which only an image (hereinafter referred to as “selected regiondisplay image”) Gsc is to be displayed in which distortion generatedthrough the image pickup optical section 111 is corrected on the imageof the image region ARs. Further, if having set the both display modeMH3, the processing control section 135 controls the image outputprocessing section 134 to generate image data DVd of a display image inwhich an entire image Gcp and a selected region display image Gsc aredisplayed simultaneously as shown in FIG. 7. Furthermore, if a pluralityof selected regions are set, it controls the image output processingsection 134 to generate image data DVd of a display image in which anentire image Gcp is displayed together with an image, as a selectedregion display image Gsc, in which distortion of highlighted imagescorresponding to these selected regions is corrected. Herein, whendisplaying the entire image Gcp and the selected region display imageGsc simultaneously as shown in FIG. 7, the image output processingsection 134 can use, for example, the on screen display (OSD)technology.

Next, the operations of the processing control section 135 will bedescribed below with reference to a flowchart of FIG. 8. The processingcontrol section 135 performs initial setting of a display mode and aregion selection mode and setting of a selected region (ST1001). Forexample, when an initial operation starts, a preset display mode orregion selection mode is set. Also, a selected region is set to apredetermined size and along a preset direction of field of view.Further, when the operation ends, information indicating the settingstate of the display mode and the region selection mode and the settingstate of the selected region may be stored and then this information maybe used when the operation starts next so that the operation can startwith being a state at the end of the previous operation.

The processing control section 135 determines whether or not the inputinformation PS supplied from the input section 12 is information thatbrings about a change in setting etc. (ST1002). If the input informationPS is information bringing about the change in the setting etc. (YES atST1002), the processing control section 135 changes the setting of adisplay mode and a region selection mode as well as a selected region inaccordance with the acquired input information PS and controls theoperation of the distortion correction processing section 133 so as toperform distortion correction processing on an image of the changedselected region. Also, it controls the selected region highlight displayprocessing section 132 so that the image of the changed selected regioncan be identified. Further, it controls the operation of the imageoutput processing section 134 so that image data DVd corresponding tothe changed mode may be generated (ST1003). Further, if the acquiredinput information PS brings about a change in size or shape of theselected region, it controls the distortion correction processingsection 133 and the selected region highlight display processing section132 so that distortion correction processing and highlight displayprocessing may be performed corresponding to the changed selectedregion. For example, if the selected region is set to the shape of acircle, an ellipsoid, a triangle, a pentagon, or a further polygon, ashape constituted of straight lines and curves, or a more complicatedgeometry, it controls the distortion correction processing section 133so that distortion correction may be performed on an image included inthe selected region having any one of those shapes. It further controlsthe operation of the selected region highlight display processingsection 132 so as to provide a highlighted image that corresponds to theselected region having any one of those shapes. It should be noted thatif the input information PS brings about no change in setting orselection (NO at ST1002), the operation goes back to ST1002 to decidewhether or not input information PS newly supplied from the inputsection 12 brings about a change in setting or selection.

Hereinafter, the processing of the processing control section 135acquiring the input information PS and obtaining an image of a selectedregion in a condition where its distortion is corrected is referred toas development processing.

Next, a state mode in which the image pickup optical section 111 isplaced will be described with reference to FIGS. 9 and 10. If a field ofview is represented as a three-dimensional space as shown in FIG. 9, aselected region can be represented on a sphere 52. It should be notedthat an angle θ with respect to arrow OA as an optical axis representsan angle of incidence.

In this case, if the image pickup optical section 111 is constituted ofa fish-eye lens having a field angle of, for example, about 180 degrees,its field of view corresponds to the hemisphere of the sphere 52.Therefore, if the image pickup optical section 111 is placed rightupward, as shown in FIG. 10(A), the upper half of the sphere providesthe field of view of the image pickup optical section 111. It should benoted that the field of view of the upper half of the sphere is referredto as an upper hemispherical field of view. Also, if the image pickupoptical section 111 is placed right downward, as shown in FIG. 10(B),the lower half of the sphere provides the field of view of the imagepickup optical section 111. It should be noted that the field of view ofthe lower half of the sphere is referred to as a lower hemisphericalfield of view. Further, if the image pickup optical section 111 isplaced horizontally to pick an image from the front side, as shown inFIG. 10(C), the front half of the sphere provides the field of view ofthe image pickup optical section 111. It should be noted that the fieldof view of the front half of the sphere is referred to as a fronthemispherical field of view. Also, if picking an image up from the rightor left side instead of the front side, a right or left hemisphericalfield of view is obtained, respectively.

In a case where the image pickup optical section 111 is placed rightupward, that is, the optical axis of the image pickup optical section111 is roughly aligned with the vertical line to direct a picking-updirection upward, such a situation is assumed that, for example, theuser may look upward from the ground, the floor, or the desk. In a casewhere the image pickup optical section 111 is placed right downward,that is, the optical axis of the image pickup optical section 111 isroughly aligned with the vertical line to direct the picking-updirection downward, such a situation is assumed that, for example, theuser may look downward from the ceiling or the sky. In a case where theimage pickup optical section 111 is placed horizontally, such asituation is assumed that, for example, the user may see from a walletc. perpendicular to the ground horizontally or laterally.

Besides, such a field of view may be through of as an obliquely upwardor downward half. In such a manner, a hemispherical field of view isobtained along a direction in which the image pickup optical section 111is placed (direction in which the image pickup section 11 is placed ifthe image pickup optical section 111 and the image pickup element 112are integrated with each other). It should be noted that not only in acase where a fish-eye lens is used but also in a case where a wide-anglelens or a mirror is used, the direction of the field of view changes inaccordance with the direction of placement. Further, if a field of viewis large, a part of its range of the field of view may be selected andthis selected part of the range of the field of view may be utilized.

Next, distortion correction processing by the distortion correctionprocessing section 133 will be described below. As a method of thedistortion correction processing, a geometrical correction technique mayas well be used to employ a general algorithm that converts, forexample, a two-dimensional coordinate system containing distortion intoa two-dimensional orthogonal coordinate system containing no distortion.In this case, a conversion equation or table may be saved in an ROM orany other memory, not shown. However, besides such distortion correctiontechnique, any other publicly known distortion correction technique maybe used.

FIG. 11 are explanatory diagrams of image height characteristics of alens. In FIG. 11(A), an upper hemispherical field of view around pointis displayed two-dimensionally as viewed in the y-axial direction. Inthe figure, an arrow OPk indicates a direction of a subject, forexample. Assuming a focal point of the subject placed in the directionindicated by this arrow OPk to be point Q, a distance from the point Oto the focal point Q provides an image height Lh. FIG. 11(B) is a graphwhich illustrates the characteristics of this image height. Itshorizontal axis represents an angle (angle of incidence) θ and itsvertical axis represents an image height Lh. Its data may be stored in amemory as a conversion table beforehand.

FIG. 12 are explanatory diagrams of the principle of the distortioncorrection processing. FIG. 12(A) is a display plane 81 which shows therange of an image to be displayed on the display section 14. FIG. 12(B)shows a state where the display plane 81 is set with respect to thesphere 51 in the upper hemispherical field of view. Herein, if an imagein a selected region is to be displayed on the display section 14, thedisplay plane 81 which shows the range of this display image correspondsto a selected region. FIG. 12(C) shows a state where the sphere 51 shownin FIG. 12(B) is projected onto an x-y plane, so that an area to whichthis sphere 51 is projected corresponds to the region of the entireimage Gcp.

A point, for example, a point P on the display plane 81 which is setwith respect to the sphere 51 of the upper hemispherical field of viewwill be described below. Assuming the position of this point P to be P(u, v, w), angle θ can be obtained by calculating θ=arccos[w/((u²+v²+w²)^(1/2))] because OP=(u²+v²+w²)^(1/2). It should be notedthat a center of the display plane is assumed to be HO. Further, byobtaining beforehand the above image height characteristics of the imagepickup optical section 111 and storing a conversion table of the angle θand the image height Lh, an image height Lh with respect to the point Pcan be obtained by calculating the angle θ.

Further, assuming an intersection point between the x-y plane and aperpendicular line dropped from the point P to the x-y plane to be pointP′ (u, v, 0), OP′=OP×sin(θ). Therefore, a focal point Q(xp, yp) takes ona position where xp=u×Lh/((u²+v²+w²)^(1/2)×sin(θ) andyp=v×Lh/((u²+v²+w²)^(1/2)×sin(θ) and thus, the focal point Q(xp, yp) canbe obtained.

Further, the position of the focal point Q may be obtained from an angleθ and an image height Lh after obtaining the angle φ between the x-axisand a direction of the point P′ on the x-y plane corresponding to thepoint P. Herein, the angle φ can be obtained by calculating φ=arccos[u/((u²+v²)^(1/2))]. Therefore, a position which is away from the pointO by the image height Lh in a direction of the angle φ with respect tothe x-axis provides the focal point Q.

By obtaining a pixel signal of the thus obtained focal point Q from theimage pickup pixel 112, point P is drawn on the display plane 81 basedon the pixel data as shown in FIG. 12D. Further, by performing the sameprocessing on every point (pixel) on the display plane 81, it ispossible to display on the display plane 81 an image whose distortiondue to the image pickup optical section 111 is corrected. It should benoted that if there is no pixel signal corresponding to the focal pointQ, the pixel signal of pixels around the focal point Q may be used togenerate a pixel signal corresponding to the focal point Q. For example,through interpolation etc. between the pixel signals of the pixelsaround the focal point Q, it is possible to generate the pixel signalthat corresponds to the focal point Q.

By thus performing the drawing through the use of the pixel data of thefocal point Q corresponding to each of the points on the display plane81, a distortion-corrected image can be obtained, so that the processingcontrol section 135 generates information that can identify a focalpoint corresponding to a selected region, that is, an image region ARscorresponding to the selected region, as the selected region settinginformation JA. For example, the processing control section 135generates information that indicates a selected region by using angles θand φ shown in FIG. 12B as the selected region setting information JA.In this case, an image region ARs corresponding to the selected regioncan be identified from an image height Lh corresponding to the angle θand the angle φ, so that the selected region highlight displayprocessing section 132 can generate image data DVcp of an entire imageGcp in which an image region ARs corresponding to the selected region isgiven as a highlighted image Gs. Further, by obtaining pixel datacorresponding to each of pixels based on angles θ and φ that indicatethose pixels on the display plane corresponding to a selected region,the distortion correction processing section 133 can generatedistortion-corrected image data DVsc of a selected region display imageGsc on which distortion correction processing is performed. Further,even if a coordinate value that indicates the range of a selected regionis used as the selected region setting information JA, by performing theabove-described calculation processing etc., it is possible to generatethe image data DVcp of an entire image Gcp and the distortion-correctedimage data DVsc of a selected region display image Gsc. Further, byusing a coordinate value, even if a selected region has a complicatedshape, the selected region can be shown easily.

Next, the region selection mode will be described below. As the regionselection mode MS, the orthogonal coordinate mode MS1 serving as thefirst region selection mode and the polar coordinate mode MS2 serving asthe second region selection mode are prepared, as shown in FIG. 4.

The orthogonal coordinate mode MS1 is a mode such that, for example, asshown in FIG. 10(C), if viewing from a wall etc. perpendicular to theground horizontally or laterally as the front hemispherical field ofview, an distortion-corrected image of a desired subject can be obtainedeasily. Specifically, if the input information PS that indicates aninstruction to switch a selected region is supplied from the inputsection 12, the processing control section 135 performs calculationprocessing to move the selected region 71 in the axial direction of anorthogonal coordinate system based on the switchover instruction,thereby generating selected region setting information JA that indicatesa newly set selected region.

FIG. 13 is an explanatory illustration of operations in the orthogonalcoordinate mode MS1 out of the region selection modes. In the orthogonalcoordinate mode MS1, the selected region 71 is switched using anorthogonal coordinate system in accordance with a switchoverinstruction. The switchover instruction indicates, for example, anx-coordinate value and a y-coordinate value of the post-switchoverselected region or x-directional and y-direction movement distances ofthe selected region, thereby switching the selected region 71 on theorthogonal coordinates. Herein, if only one of the x-coordinate and they-coordinate is changed, the selected region 71 moves to a new positionalong the axial direction of the orthogonal coordinate system. Further,if the x-coordinate and the y-coordinate are changed, the selectedregion 71 moves to a new position obliquely with respect to the axialdirection of the orthogonal coordinate system.

If the selected region 71 is sequentially set by moving it in the x-axisdirection based on a switchover instruction, the trajectory of anarbitrary point (for example, center PO) in the selected region 71follows line 51 x. On the other hand, if the selected region 71 issequentially set by moving it in the y-axis direction based on aswitchover instruction, the trajectory of center PO in the selectedregion 71 follows line 51 y. It should be noted that as the selectedregion 71 moves, the image region ARs also moves.

In such a manner, in the orthogonal coordinate mode MS1, theabove-described selected region is switched by changing the coordinatevalues of the orthogonal coordinate system. Therefore, by selecting theorthogonal coordinate mode MS1 in a case where the front hemisphericalfield of view is provided, the selected region 71 can be easily set to ahorizontally or vertically moved position in accordance with a selectedregion switchover instruction, so that an image displayed on the displaysection 14 can be easily switched to an image present in a desireddirection. For example, it is easily possible to select a desired one ofhorizontally arranged subjects and display the same.

Next, the polar coordinate mode MS2 will be described below. The polarcoordinate mode MS2 is a mode such that, for example, as shown in FIG.10(A), if viewing upward from the ground, a floor, or a desk as theupper hemispherical field of view or, as shown in FIG. 10B, if viewingdownward from the ceiling or the sky as the lower hemispherical field ofview or the like, an image of a desired subject on which distortioncorrection is performed can be obtained easily. Specifically, if theinput information PS that indicates an instruction to switch a selectedregion is supplied from the input section 12, the processing controlsection 135 performs calculation processing to move the selected region71 in a direction in which an argument is changed of the polarcoordinate system based on the switchover instruction, therebygenerating selected region setting information JA that indicates a newlyset selected region.

FIG. 14 is an explanatory illustration of operations in the polarcoordinate mode MS2 out of the region selection modes. In the polarcoordinate mode MS2, the selected region 71 is switched using a polarcoordinate system in accordance with a switchover instruction. If afield of view is expressed as a three-dimensional space as shown in FIG.14, for example, a switchover instruction indicates an argument θag andan argument φag of the post-switchover selected region and a variationangle of the argument θag or that of the argument φag so that theselected region 71 may be switched in the polar coordinates. Herein, ifonly any one of the argument θag and the argument φag is changed, theselected region 71 moves to a new position in a direction in which theargument θag of the polar coordinate system is changed (hereinafterreferred to as “θag-changed direction) or a direction in which theargument φag is changed (hereinafter referred to as “φag-changeddirection) thereof. Further, if the argument θag and the argument φagare changed, the selected region 71 moves to a new position obliquelywith respect to the θag-changed direction or the φag-changed directionof the polar coordinate system.

If the selected region 71 is sequentially set by moving it in theθag-changed direction based on a switchover instruction, the trajectoryof an arbitrary point (for example, center PO) in the selected region 71follows line 51 r. On the other hand, if the selected region 71 issequentially set by moving it in the φag-changed direction based on aswitchover instruction, the trajectory of center PO in the selectedregion 71 follows line 51 s. It should be noted that as the selectedregion 71 moves, the image region ARs also moves.

Further, if a field of view is expressed using an image which isexpressed in a two-dimensional space, a switchover instructionindicates, for example, an argument φag and a moving radius of thepost-switchover selected region or a variation angle of the argument φagor that of the moving radius, thereby switching the selected region 71in polar coordinates also in the case of expressing the field of viewusing a two-dimensional space.

In such a manner, in the polar coordinate mode MS2, the selected regionis switched by changing the argument and/or the moving radius of thepolar coordinate system. Therefore, by selecting the polar coordinatemode MS2 in a case where the upper or lower hemispherical field of viewis provided, the selected region 71 can be easily set to a positionreached by moving in an argument-changed direction in accordance with aselected region switchover instruction, so that an image displayed onthe display section 14 can be easily switched to an image present in adesired direction. For example, it is easily possible to select adesired one of subjects arranged around the image pickup optical section111 and display the same.

FIG. 15 shows one example of a graphical user interface (GUI) which isdisplayed when a user operates a selected region using the input section12. Operation input screens Gu shown in FIGS. 15(A) and 15(B) may bedisplayed on the display section 14 together with the entire image Gcpand the selected region display image Gsc which have been shown in FIG.7. Alternatively, the operation input screens Gu and the entire imageGcp etc. may be separately displayed on different display sections.Further, a display section separate from the display section 14 on whichthe entire image Gcp and the selected region display image Gsc aredisplayed may be provided at, for example, the input section 12, thusproviding GUI display on this display section. The operation inputscreen Gu is provided with a direction button group Gua or a directionbutton group Gub, a “Scale up” button Guc1, and a “Scale down” buttonGuc2. In FIG. 15(A), as the direction button group Gua, for example,there are a central “Select” button Gua1 and the other direction buttonsGua2 such as “Up”, “Down”, “Right”, and “Left” buttons surrounding it.Further, in FIG. 15(B), as the direction button group Gub, for example,there are a central “Select” button Gub1 and the other bearing buttonsGub2 such as “North” and “SE (South east)” buttons surrounding it.

FIG. 16 are explanatory illustrations of a case where it is instructedto switch a selected region when the orthogonal coordinate mode MS1 isselected. In this example, a case in which the both display mode MH3 isemployed as the display mode MH is described. Further, in the followingdescription, an example is used in which the user perform operations asviewing the operation input screen Gu etc. shown in FIG. 15(A).

Suppose that the user presses the “Right” button out of the directionbuttons Gua2 once or a plurality of times or continuously by using themouse or the keyboard etc. in a state shown in FIG. 16(A). It should benoted that “to press continuously” refers to a state in which a button,if pressed once, is held down without being released, that is“continuous pressing”.

If the “Right” button is pressed in such a manner, in accordance withthe corresponding input information PS input through the input section12, the processing control section 135 in the image processing section13 performs switchover processing on a selected region in a directionthat corresponds to the orthogonal coordinate mode MS1, therebygenerating selected region setting information JA that indicates a newlyset selected region. Further, the processing control section 135supplies the generated selected region setting information JA to theselected region highlight display processing section 132 to change ahighlight display region so that a highlighted image Gs may show animage region ARs that corresponds to the newly set selected region.Moreover, the processing control section 135 supplies the generatedselected region setting information JA to the distortion correctionprocessing section 133 to provide the image of an image region ARscorresponding to the newly set selected region as an selected regiondisplay image Gsc in which distortion generated through the image pickupoptical section 111 is corrected.

Further, the image output processing section 134 generates image dataDVd of a display image including the selected region display image Gscin accordance with the display mode and supplies it to the displaysection 14.

Thus, in the display section 14, as shown in FIG. 16(B), an image inwhich the selected region is moved rightward is displayed. In the entireimage Gcp also, the position of the highlighted image Gs is updated.Further, if the “Right” button is pressed a plurality of number of timesor continuously, the processing control section 135 sets a movementdistance for the selected region in accordance with the number of timesthe “Right” button is operated or a period of time when it is held down,thereby updating the selected region display image Gsc. In such amanner, the processing control section 135 performs the developmentprocessing in the orthogonal coordinate system, to display the selectedregion display image on the display section 14.

Further, if the “Up” button is pressed by the user, in accordance withthe corresponding input information PS, the processing control section135 performs switchover processing on a selected region, to cause thedisplay section 14 to display the highlighted image Gs of an imageregion ARs which moves as the selected region is switched, as a selectedregion display image Gsc in which distortion generated through the imagepickup optical section 111 is corrected.

The respective “Select” buttons Gua1 and Gub1 in FIGS. 15(A) and 15(B)may possibly be utilized variously. For example, they can be utilized asa recording start button to record a selected region display image Gscin the current image region ARs. Further, in a case where an imageregion ARs is not displayed as a highlighted image Gs, they can beutilized as a region selection operation start button to start a regionselection operation so that the user can select a region in a conditionwhere this image region ARs is displayed as a highlighted image Gs.Alternatively, they can be utilized as a switchover button of thedisplay mode MH or any one of various other determination buttons.

Further, in a case where a reference position at the image pickupsection 11 is properly set in a predetermined direction, for example, inthe northern direction, if the processing control section 135 determinesthat the “East” button out of the direction buttons Gu2 is operated, theprocessing control section 135 newly sets a selected region in the“East” direction, thereby updating the selected region display imageGsc. Further, if the processing control section 135 determines that the“West” button is operated, it newly sets the selected region in the“West” direction, thereby updating the selected region display imageGsc. In such a manner, by operating the button that indicates a desireddirection, an image having the selected region set to a desired newposition can be displayed on the display section 14 without generatingdistortion.

It should be noted that GUI displays shown in FIGS. 15(A) and 15(B) arejust illustrative and, of course, not limited to them.

FIG. 17 show display images which are displayed on the display section14 when the display modes are sequentially switched if, for example, theorthogonal coordinate mode is selected. FIG. 17(A) shows a display imagein a case where the entire image display mode MH1 is set, displayingonly an entire image Gcp. FIG. 17(B) shows a display image in a casewhere the selected image display mode MH2 is set, displaying only aselected region display image Gsc. FIG. 17(C) shows a display image in acase where the both display mode MH3 is set, displaying both of anentire image Gcp and a selected region display image Gsc. Further, ifthose display modes are switched from each other sequentially, theentire image display mode MH1, the selected image display mode MH2, andthe both display mode MH3 are switched cyclically.

FIG. 18 show display images which are displayed on the display section14 when the polar coordinate mode MS2 is selected. Further, FIG. 18 showa case where two selected regions are provided. It should be noted thatthe number of the selected regions is not limited to one or two but canbe set arbitrarily by the user. For example, that number may beincremented each time a “Menu” button Guc3 shown in FIG. 15(A) or 15(B)is operated or any other GUI display, not shown, may be used. In thiscase, the processing control section 135 performs processing to providea plurality of selected regions in accordance with input information PSfrom the input section 12. Alternatively, irrespective of the inputinformation PS from the input section 12, a plurality of predeterminedselected regions may be previously provided in setting. Further, if thedivided display mode MH4 is set, in accordance with the inputinformation PS from the input section 12, the processing control section135 may generate the selected region display images of the selectedregions individually and display them on the display section 14simultaneously.

Now, the description of the polar coordinate mode MS2 will be assumedbelow. For simplification of the description, such a state that a robotmay go through a piping which is so small that a human being cannotenter is assumed in FIG. 18. Further, this robot is supposed to befitted with the image pickup optical section 111 constituted of, forexample, a wide-angle lens. Further, the piping is supposed to have acrack 92 formed at the upper part of its inner wall 91. A selectedregion display image Gsc1 is the first selected region image. That is,it is an image in which distortion correction processing is performed onthe highlighted image Gs1 of an image region ARs1 that corresponds tothe first selected region. Further, a selected region display image Gsc2is an image in which distortion correction processing is performed onthe highlighted image Gs2 of an image region ARs2 that corresponds tothe second selected region.

In the state of FIG. 18(A), if the input information PS supplied fromthe input section 12 is a switchover instruction to switch a selectedregion in a φag-changed direction of the polar coordinate system shownin FIG. 14, for example, the processing control section 135 performsprocessing to switch the selected region in the φag-changed direction inaccordance with the switchover instruction and supplies the selectedregion setting information JA that indicates the post-processingselected region to the selected region highlight display processingsection 132 and the distortion correction processing section 133. Theselected region highlight display processing section 132 displayshighlighted images Gs1 and Gs2 that correspond to the post-processingselected region based on the selected region setting information JA.Based on the selected region setting information JA, the distortioncorrection processing section 133 corrects the images of image regionsARs1 and ARs2 that correspond to the post-processing selected regioninto the selected region display images Gsc1 and Gsc2 in a conditionwhere distortion generated through the image pickup optical section 111is corrected. Therefore, as shown in FIG. 18(B), display images afterthe switchover instruction is issued are the images of the selectedregion after the switchover instruction is issued that will be displayedas the selected region display images Gsc1 and Gsc2. Further, thehighlighted images Gs1 and Gs2 properly indicate the regions of theselected region display images Gsc1 and Gsc2. In this case, the imageregions ARs1 and ARs2 have been moved counterclockwise on the entireimage Gcp and, if the switchover instruction is of an oppositedirection, have been moved clockwise direction.

Further, in the state of FIG. 18(A), if the input information PSsupplied from the input section 12 is a switchover instruction to switcha selected region in a θag-changed direction of the polar coordinatesystem shown in FIG. 14, for example, the processing control section 135performs processing to switch the selected region in the θag-changeddirection in accordance with the switchover instruction and supplies theselected region setting information JA that indicates thepost-processing selected region to the selected region highlight displayprocessing section 132 and the distortion correction processing section133. The selected region highlight display processing section 132displays highlighted images Gs1 and Gs2 that correspond to thepost-processing selected region based on the selected region settinginformation JA. Based on the selected region setting information JA, thedistortion correction processing section 133 corrects the images ofimage regions ARs1 and ARs2 that correspond to the post-processingselected region into the selected region display images Gsc1 and Gsc2 ina condition where distortion generated through the image pickup opticalsection 111 is corrected. Therefore, as shown in FIG. 19, display imagesafter the switchover instruction is issued are the images of theselected region after the switchover instruction is issued that will bedisplayed as the selected region display images Gsc1 and Gsc2. Further,the highlighted images Gs1 and Gs2 properly indicate the regions of theselected region display images Gsc1 and Gsc2. In this case, the imageregions ARs1 and ARs2 have been moved to be closer to each otherradially on the entire image Gcp and, if the switchover instruction isof an opposite direction, have been moved to be more away from eachother on the entire image Gcp.

It should be noted that in the polar coordinate mode MS2, for example, aselected region can be switched using the mouse, the keyboard, the touchsensor, etc., in which case the GUI may be of any form.

In the polar coordinate mode MS2, for example, with respect to a stateof the display image shown in FIG. 18(A), as shown in FIG. 20, it ispossible to display the selected region display image Gsc1 at the lowerpart of the display section 14 in a condition where it is rotateddownward by 180 degrees and the selected region display image Gsc2 atthe upper part of the display section 14 in a condition where it isrotated upward by 180 degrees in accordance with the input informationPS acquired from the input section 12. In such a manner, convenientlythe user can view the image at an easy-to-see angle.

Of course, in the above-described orthogonal coordinate mode MS1 also, aplurality of selected regions is provided, so that the divided displaymode MH4 may be applied correspondingly. Alternatively, in theorthogonal coordinate mode MS1 and the polar coordinate mode MS2, evenif a plurality of selected regions are provided, the image processingsection 13 generates a selected region display image of one of theselected region rather than generating a combined distortion-correctedimage of one screen and output it to the display section 14. In thiscase, whether or not to output image data that displays the selectedregion display image of the one selected region on one screen or togenerate the selected region display images of the plurality of selectedregions and display those selected region display images on one screenis controlled by the processing control section 135 in the imageprocessing section 13 in accordance with the input information PS fromthe input section 12 or predefined setting information.

In the polar coordinate mode MS2, for example, each time the “Menu”button Guc3 is operated by the user, the processing control section 135switches the display mode MH similar to the orthogonal coordinate modeMS1. Furthermore, if a plurality of selected regions is provided, thedivided display mode MH4 can be selected.

FIG. 21 is an illustration showing transition of the change of thedisplay modes in a case where selection of the divided display mode MH4is enabled. It is here supposed that four selected regions are provided.For example, in display, it is possible to switch a mode (FIG. 21(A)) inwhich one selected region corrected image is displayed in the bothdisplay mode MH3, a mode (FIG. 21(B)) in which one selected regiondisplay image is displayed in a condition where it is turned upside downin the both display mode MH3, a mode (FIG. 21C) in which two selectedregion corrected images are displayed in the divided display mode MH4, amode (FIG. 21(D)) in which images in which two selected region correctedimages are displayed is displayed in a condition where they are turnedupside down in the divided display mode MH4, a mode (FIG. 21(E)) inwhich four selected region corrected images are displayed in the divideddisplay mode MH4, and a mode (FIG. 21(F)) in which images in which fourselected region corrected images are displayed are displayed in acondition where they are turned upside down in the divided display modeMH4. It should be noted that since not all of the selected regioncorrected images are displayed in FIG. 21, the selected region correctedimage to be displayed may be switched. It should be noted that it is ofcourse permitted to enable switchover to the entire image display modeMH1 and the selected image display mode MH2.

In such a manner, according to the present embodiment, since theorthogonal coordinate mode MS1 and the polar coordinate mode MS2 areprovided as the region selection mode, for example, by switching theregion selection mode in accordance with the direction of a field ofview, it is possible to perform intuitive and easy-to-understandoperations and to realize an image processing apparatus convenient andeasy-to-use for the user.

Besides the above-described processing, the image processing section 13can perform processing to scale up or down and rotate a selected regiondisplay image.

The processing to scale up/down a selected region display image will bedescribed below with reference to FIG. 22. In a state shown in FIG.22(A), the user operates a “Scale up” button Guc1 shown in FIGS. 15(A)and 15(B), for example. Then, the processing control section 135performs processing to scale up the range of a selected region inaccordance with the current input information PS and supplies theselected region setting information JA that indicates thepost-processing selected region to the selected region highlight displayprocessing section 132 and the distortion correction processing section133. The selected region highlight display processing section 132displays highlighted images Gs1 and Gs2 that corresponds to thepost-processing selected region based on the selected region settinginformation JA. The distortion correction processing section 133performs correction processing to correct the images of the imageregions ARs1 and ARs2 that correspond to the post-processing selectedregion to the selected region display images Gsc1 and Gsc2 in which thedistortion generated through the image pickup optical section 111 iscorrected based on the selected region setting information JA. Herein,if the both display mode MH3 is provided as shown in FIG. 22(A), theselected region display image Gsc that corresponds to the scaled-downselected region is displayed on the entire screen except for the displayregion of the entire image Gcp, as shown in FIG. 22B, so that thepost-scale-down display image is displayed so as to enlarge a personimage GM included in the image region ARs as compared to FIG. 22A. Itshould be noted that since the selected region is scaled down, the imageregion ARs in the entire image Gcp becomes smaller.

Conversely, if the user operates a “Scale down” button Guc2, theprocessing control section 135 performs processing to scale down therange of a selected region in accordance with the current inputinformation PS and supplies the selected region setting information JAthat indicates the post-processing selected region to the selectedregion highlight display processing section 132 and the distortioncorrection processing section 133. Therefore, the highlighted image Gsof an image region ARs that corresponds to the post-processing selectedregion has its distortion corrected and is displayed as a selectedregion display image Gsc on the entire screen except for the displayregion of the entire image Gcp, so that the person image GM included inthe image region ARs is displayed so as to reduce as compared to FIG.22A. In such a manner, the scale up/down processing can be performed.

Next, a case in which the image of a selected region rotates by theimage processing section 13 and is displayed will be described belowwith reference to FIGS. 23 and 24. FIGS. 23(A) and 23(B) show a statebefore a “Rotation” operation is instructed, out of which FIG. 23(A)shows a case where a selected region is set in such a manner that, forexample, the person image GM in the entire image Gcp may be included inan image region ARs. On the other hand, FIG. 23(B) shows a selectedregion display image Gsc, which is an image obtained by performingdistortion correction processing on a highlighted image Gs of the imageregion ARs. If the user issues a “Rotation” operation instruction, asshown in FIG. 24(A), the processing control section 135 performsalteration processing on the selected region so that it may rotatearound a roughly central point of the image region ARs in accordancewith the input information PS. In this case, in the image region ARsthat corresponds to the post-alteration selected region, the personimage GM is rotated in a reverse direction. Therefore, by generating aselected region display image Gsc for the highlighted image Gs of theimage region ARs that corresponds to the post-alteration selectedregion, it is possible to obtain an image in which the person image GMis rotated in a rotation direction opposite to the rotation direction ofthe selected region as shown in FIG. 24(B). Through such rotationprocessing, the user can view an observation target at an easy-to-seeangle conveniently.

Further, instead of rotating the selected region, the entire image Gcpmay be rotated. For example, from a state in which the x-axis (pan axis)becomes a horizontal direction and the y-axis (tilt axis) becomes avertical direction as moving axes in the orthogonal coordinate mode MS1as shown in FIG. 25(A), the entire image Gcp is rotated, for example,counterclockwise together with these x-axis and y-axis as well as thehighlighted image Gs as shown in FIG. 25(B). In this case, there is nochange caused in the selected region display image Gsc which is providedto the display section 14. It is thus possible to correct theinclination of a camera placing angle in the above-described rotationdirection or intentionally rotate the entire image Gcp and display it asspecial effects in the entire image display mode MH1.

FIG. 26 is a block diagram showing the constitution of an imageprocessing system according to another embodiment of the presentinvention. Hereinafter, the description of the devices, the functions,etc. similar to those of the image processing system 10 shown in FIG. 1will be simplified or omitted to concentrate on the different respects.

The image processing system 20 has such a constitution as to add astorage apparatus 21 to the image processing system 10 shown in FIG. 1.The storage apparatus 21 is an apparatus for storing, for example, imagedata DVa generated by an image pickup section 11 and a variety of kindsof image data generated by an image processing section 13. As a storagedevice used in the storage apparatus 21, such a device may be used thatcan store image data such as an optical disk, a magnetic disk, asemiconductor memory, a dielectric memory, a tape-like storage medium.

If image data DVa is stored in the storage apparatus 21, for example,the image processing section 13 can read the image data DVa desired by auser from the storage apparatus 21 in accordance with input informationPS from an input section 12 and display it on a display section 14.Specifically, such an aspect can be thought of that in accordance withthe input information PS based on a user operation, the image processingsection 13 may read the image data DVa of a past wide-field image Gcstored in the storage apparatus 21, set a selected region for a field ofview represented by this read image data DVa, and display on the displaysection 14 a selected region display image, which is an image in whichthe distortion of this selected region is corrected. Alternatively, suchan aspect may be considered that independently of the user, the imageprocessing section 13 may perform distortion correction processing on animage of a predefined selected region out of past wide-field imagesstored in the storage apparatus 21 and display it on the display section14.

In this case, as a specific example, the following aspect may be thoughtof. For example, suppose that the user selects a selected region in awide-field image obtained in a real time from the image pickup section11 and views an entire image Gcp and a selected region display image Gscin a real time or stores them in the storage apparatus 21. Then, theuser can also select a region different from the above region selectedin a real time and view its selected region display image Gsc whileviewing the stored entire image Gcp.

Alternatively, instead of storing the image data of the entire image Gcpin the storage apparatus 21, the image processing section 13 can alsostore only the image data of the selected region display image Gsc. Inthis case, the user can view that selected region display image Gsclater. Of course, it may store the image data that indicates both of theentire image Gcp and the selected region display image Gsc or the imagedata of each of those images.

Alternatively, the image processing section 13 can also perform theprocessing of a flowchart shown in FIG. 27. The image processing section13 acquires a real-time wide-field image from the image pickup section11 (ST2401) and also acquires a past wide-field image or a past selectedregion display image stored in the storage apparatus 21 (ST2402). Theimage processing section 13 can perform processing to combine the thusacquired wide-field image and past wide-field image or the past selectedregion display image into one-screen image data (ST2403) and output thecombined image data to the display section 14 (ST2404). Alternatively,the image processing section 13 may display the acquired wide-fieldimage and the past wide-field image or the past selected region displayimage on the different display sections 14. It should be noted that suchan aspect can be thought of that ST2401 and ST2402 may be reversed inorder.

Alternatively, such an aspect can be thought of that the imageprocessing section 13 may output both of the selected region displayimage Gsc (which may be an image generated in a real time from areal-time wide-field image or an image generated in a real time from apast wide-field image stored in the storage apparatus) obtained throughreal-time distortion correction processing and the past selected regiondisplay image. Specifically, as shown in FIG. 28, the image processingsection 13 acquires the wide-field image in a real time from the imagepickup section 11 or the past wide-field image from the storageapparatus 21 (ST2501). Further, the image processing section 13 acquiresa past selected region display image stored in the storage apparatus 21(ST2502). The image processing section 13 performs distortion correctionprocessing on the image of a selected region in the above-describedwide-field image acquired at ST2501 (ST2503). The image processingsection 13 performs processing to combine the selected region displayimage generated by this distortion correction processing and theselected region display image acquired at ST2502 into one-screen imagedata (ST2504) and outputs it as a selected region display image Gsc tothe display section 14 (ST2505). It should be noted that such an aspectcan be thought of that ST2501 and ST2502 may be reversed in order orST2502 and ST2503 may be reversed in order.

In the case of processing shown in FIG. 28, further, the imageprocessing section 13 can also output a selected region display image(hereinafter referred to as real-time selected region display image)obtained by performing distortion correction processing at ST2503 and apast selected region display image in such a manner that a human beingcan distinguish between them on the display section. Specifically, itcan be thought of that the image processing section 13, for example,generates an image in which an identifier is fitted to at least one ofthe real-time selected region display image and the past selected regiondisplay image or generates a frame that encloses both of these images sothat an image having a changed color of this frame may be generated.

It should be noted that if image data stored in the storage apparatus 21is of a moving image, a certain capacity of moving image data may bestored in the storage apparatus 21 in accordance with its storagecapacity so that the least recent image frames would be automaticallyerased sequentially.

Further, the following aspect of using the storage apparatus 21 can bethought of. For example, the user performs operations to set a selectedregion in a real-time wide-field image or a wide-field image which isbased on image data read from the storage apparatus 21 at an arbitrarypoint in time and, corresponding to the operations, the image processingsection 13 then stores only position information that indicates how theselected region is set in the storage apparatus 21. Further, if aselected region is switched in the above-described region selection modeMS, the image processing section 13 may store, in the storage apparatus21, trajectory information that makes it possible to reproduce theswitchover of the selected region. FIG. 29 is a flowchart showingprocessing to store the position information or the trajectoryinformation. The image processing section 13 acquires input informationwhich is used when setting a selected region in a wide-field imageacquired in a real time from the image pickup section 11 or a wide-fieldimage which is based on image data read from the storage apparatus 21(ST2601). The processing control section 135 in the image processingsection 13 generates the position information of that selected region inaccordance with the input information or, if performing switchingoperation on the selected region, generates the trajectory informationthat makes it possible to reproduce the switchover of the selectedregion (ST2602). The processing control section 135 in the imageprocessing section 13 stores the generated position information or thetrajectory information in the storage apparatus 21 (ST2603).

Such an aspect is effective in a case where an image of a predeterminedrange or an image due to a trajectory in a range in, for example, acertain place is required. For example, in a case where if the imagepickup section 11 is mounted on a fixed-point security camera, it can bethought of that an image of a predetermined range or an image of atrajectory in a range in a wide-field image is required. In this case,by setting a selected region corresponding to a predetermined range in awide-field image or the trajectory of the predetermined range, the usercan always monitor a selected region display image of that range or itstrajectory on the display section 14 as a display image. In such anaspect, for example, in a case where the storage apparatus 21 stores theselected region display image of the “Trajectory” of the predeterminedrange, the continuous movement of the selected region from its startingpoint to its ending point automatically may be repeated periodically.Further, an image for each period may be stored in the storage apparatus21. It should be noted that, of course, the present aspect is notlimited to the security purposes.

Just to make sure, the above-described predetermined range image mightas well be a still image or a moving image. The image of the trajectoryof the certain range might also as well be a still image at a positionalong this trajectory and can be stored as a moving image covering fromthe starting point to the ending point of the trajectory. In this case,the image processing section 13 may possibly perform processing tooutput a post-distortion-correction image as a still image.

FIG. 30 is an explanatory diagram of a method of setting such atrajectory of a predetermined range, as described above, as an aspect ofusing the storage apparatus 21.

The user sets a selected region to a wide-field image to switch theposition of the selected region. Specifically, this is realized byrepeating processing such that it is determined which one of thedirection button Gua2 and the bearing position Gub2 shown in FIGS. 15(A)and 15(B), for example, is operated and the selected region is moved inthe direction indicated by the operated button; in the internalprocessing of the image processing system 20, as shown by ST2602 andST2603 of FIG. 29, the image processing section 13 stores the positioninformation of the current selected region in the storage apparatus 21in accordance with the input information that indicates the operationsof the “Select” buttons Gua1 and Gub1. For example, if the “Select”button Gua1 or Gub1 is operated when an image region ARs correspondingto a selected region is located at position a on a wide-field image Gc,it stores the position information of the selected region at this pointof time. Further, if the “Select” button Gua1 or Gub1 is operated whenthe selected region is switched and an image region ARs corresponding tothe post-switchover selected region is located at position b on thewide-field image Gc, it stores the position information of the selectedregion at this point of time. Similarly, it stores the positioninformation of the selected region at a point of time when the imageregion ARs corresponding to the selected region is located at positionsc and d on the wide-field image Gc.

Alternatively, even if the user does not operate the direction buttonGua2 or the bearing button Gub2, a trajectory may have been set using aprogram beforehand or generated though automatic recognition by use ofthe above-described various sensors. In this case, the trajectory may besuch that a selected region would be set so as to set an image regionARs to discrete points such as positions a, b, c, and d shown in FIG. 30or that a selected region might be set continually from points a throughd. Alternatively, if the user has set the discrete points of positionsa, b, c, and d, the image processing section 13 may have such a programinstalled in it as to set a selected region so that an image region ARsmay be set to a position for interpolating the points of positions a, b,c, and d.

Alternatively, a plurality of pieces of trajectory information may beprovided beforehand so that the user can select any one of them.

FIG. 31 is a block diagram showing a constitution of an image processingsystem according to further embodiment of the present invention. Thisimage processing system 30 is provided with a storage apparatus 21, notthe above-described image pickup section 11. In the storage apparatus21, the wide-field images are stored beforehand as described above, forexample. Such a constitution enables an image processing section 13 toread the image data DVm of the wide-field images and obtain a selectedregion display image from that wide-field image by performingdevelopment processing.

Besides, in the image processing system 30, the image processing section13 can obtain a selected region display image through developmentprocessing from a wide-field image which is based on image data storedin the storage apparatus 21 beforehand and then store those wide-fieldimage and selected region display image in the storage apparatus 21 in acondition where they are correlated with each other. Alternatively, theimage processing section 13 can also store a wide-field image which isbased on image data stored beforehand and information indicating aselected region on which development processing is to be performed fromthat wide-field image in the storage apparatus 21 in a condition wherethey are correlated with each other.

The present invention is not limited to the above-described embodimentsbut can be modified variously.

It is also possible to alternately display an entire image Gcp and aselected region display image Gsc to be displayed on the display section14 on the display section 14 every predetermined lapse of time. In thiscase, in response to any input operation of the user, an entire imageGcp and a selected region display image Gsc may both be displayed.

In FIGS. 1 and 26, an image pickup element 112 in the image pickupsection 11, an input section 12, an image processing section 13, thedisplay section 14, the storage apparatus 21, etc. may be connected toeach other through the Internet, a local area network (LAN) or othernetwork such as a dedicated line.

The image processing systems according to the above-describedembodiments can be applied in various fields such as, for example, asecurity system, a teleconference system, a system for inspecting,managing, and testing machinery and facilities, a road traffic system, asystem that uses a moving camera (for example, camera for photographingfrom a moving vehicle, aircraft, or any other movable bodies), anursing-care system, a medical system, etc.

it is also possible to realize an image processing system that combinesthe embodiments shown in FIGS. 26 and 31 respectively. That is, such asystem can be realized as to include the storage apparatus at each ofits front stage and rear stage.

Next, still further embodiment of the present invention will bedescribed below. In the present embodiment, a processing control section135 in an image processing section 13 can switch a region selection modeMS in accordance with an installation direction (installation angle) inwhich an image pickup section 11 is installed. The following willdescribe a constitution and an operation of an image processing system40 in this case.

FIG. 32 is a block diagram showing a constitution of the imageprocessing system according to the still further embodiment of thepresent invention. As shown in FIG. 32, in the present embodiment, inaddition to the components of the image processing system 10 shown inFIG. 1, the image processing system 40 has a direction detection sensor,for example, a gyro-sensor 41 that detects a direction in which anoptical image is made incident upon a image pickup optical section 111.

The gyro-sensor 41, which is fixed to the image pickup optical section111, detects a direction in which an optical image is made incident uponthe image pickup optical section 111, and supplies a sensor signal ESindicative of a result of the direction detection to the processingcontrol section 135 in the image processing section 13. It should benoted that the following description is based on the assumption that theimage pickup optical section 111, the image pickup element 112, and thegyro-sensor 41 are integrated into the image pickup section 11.

FIG. 33 are conceptual illustrations of a manner to switch a regionselection mode MS in accordance with a direction in which an imagepickup section 11 is placed in the present embodiment.

As shown in FIG. 33, states in which the image pickup section 11 isplaced may be roughly divided into three cases: as shown in the aboveFIG. 10, a case where it is placed on the ground, a floor, a desk, etc.to pick up a desired subject MH (upper hemispherical field of view inFIG. 33(A)), a case where it is placed on a wall etc. perpendicular tothe ground to pick up the subject MH (front hemispherical field of viewin FIG. 33(B)), and a case where it is fixed on or hung from a ceilingto pick up the subject MH (lower hemispherical field of view in FIG.33(C)).

Therefore, the processing control section 135 in the image processingsection 13 automatically sets or switches the region selection mode MSin accordance with an angle in a vertical direction on the image pickupsection 11 which is determined on the basis of the sensor signal fromthe direction detection sensor.

Specifically, if the image pickup section 11 is in the upper or lowerhemispherical field of view, the region selection mode MS is switched toa polar coordinate mode MS2, and if it is in the front hemisphericalfield of view, the region selection mode MS is switched to an orthogonalcoordinate mode MS1. By thus switching the mode, if the image pickupsection 11 is in the upper or lower hemispherical field of view, it ispossible to easily and evenly observe the surroundings of a subject atthe center of a wide-field image Gc more than the subject itself.Further, if the image pickup section 11 is in the front hemisphericalfield of view, by setting the orthogonal coordinate mode MS1, it ispossible to easily observe the up-and-down direction and theright-and-left direction also of the subject with observing this subjectin detail at the center of the wide-field image Gc.

It should be noted that in the present embodiment, a state in which theimage pickup section 11 is placed in the upper hemispherical field ofview and the region selection mode MS is switched to the polarcoordinate mode MS2, as shown in FIG. 33(A), is referred to as S₀, astate in which the image pickup section 11 is placed in the fronthemispherical field of view and the region selection mode MS is switchedto the orthogonal coordinate mode MS1, as shown in FIG. 33(B), isreferred to as S₁, and a state in which the image pickup section 11 isplaced in the lower hemispherical field of view and the region selectionmode MS is switched to the polar coordinate mode MS2, as shown in FIG.33(C), is referred to as S₂.

FIG. 34 are explanatory illustrations of a method of setting thresholdvalues for switching the states S₀, S₁, and S₂. As shown in FIG. 34, inthe present embodiment, for example, supposing a state where the imagepickup section 11 is in the upper hemispherical field of view to be areference position (ψ=0 degree), first, two threshold values (ψ₁, ψ₂)are used to thereby determine any one of the above-described threestates S₀, S₁, and S₂ in accordance with whether or not the installationangle of the image pickup section 11 exceeds the threshold value ψ₁ orψ₂ and, based on the result of the decision, set the region selectionmode MS. Further, if the installation angle of the image pickup section11 is changed after the region selection mode MS has been set, athreshold value other than the above-described threshold values of ψ₁and ψ₂ is used to give a hysteresis when those states are switched fromeach other.

Specifically, as shown in FIG. 34(A), values in a ±10-degree range of,for example, the threshold value ψ₁ are set as new threshold values ψ₃and ψ₄ besides threshold values ψ₁ and ψ₂ and values in a ±10-degreerange of, for example, the threshold value ψ₂ are set as new thresholdvalues ψ₅ and ψ₆. As shown in FIG. 34(B), the threshold value ψ₃ is athreshold value when the state is switched from state S₀ to state S₁,and the threshold value ψ₄ is a threshold value when the state isswitched from state S₁ to state S₀. Further, threshold value ψ₅ is athreshold value when the state is switched from state S₁ to state S₂,and threshold value ψ₆ is a threshold value when the state is switchedfrom state S₂ to state S₁. The magnitude relation of those thresholdvalues including threshold values ψ₁ and ψ₂ become as ψ₄<ψ₁<ψ₃<ψ₆<ψ₂<ψ₅as shown in FIG. 34(B). It should be noted that although, for example,the threshold value ψ₁ is 45 degrees and the threshold value ψ₂ is 135degrees, they are not limited to those values. Further, although theabove-described threshold values ψ₃, ψ₄, ψ₅, and ψ₆ are not limited tothose in the above-described ±10-degree ranges, they may be in a±5-degree range, ±15-degree range, etc. and, further, may be set so thatdifferences between the threshold value ψ₁ and the threshold values ψ₃and ψ₄ have different absolute values or that differences between thethreshold value ψ₂ and the threshold values ψ₅ and ψ₆ have differentabsolute values.

Next, operations to switch the above-described states S₀, S₁, and S₂from each other in the present embodiment will be described below. FIG.35 is a flowchart showing the operations in a case where the imageprocessing system 40 switches states S₀, S₁, and S₂ from each other.

As shown in FIG. 35, first, the image pickup section 11 is located tothe above-described reference position (ψ=0 degree). The processingcontrol section 135 in the image processing section 13 acquires a resultof measurement by the gyro-sensor 41 at this point of time (ST3201).Next, the image pickup section 11 is located to a desired position. Theprocessing control section 135 acquires a result of measurement by thegyro-sensor 41 at this point of time and determines a present angleψ_(P) from this measurement result and that acquired at ST3201 (ST3202).

Subsequently, the processing control section 135 determines whether ornot the present angle ψ_(P) is equal to or less than the threshold valueχ₁ (ST3203). If ψ_(P)≦ψ₁ (YES), the processing control section 135determines that the above-described state S₀ is set and sets the regionselection mode MS to the polar coordinate mode MS2 in the upperhemispherical field of view (ST3204).

If ψ_(P)>ψ₁ (NO) at the above step of ST3203, the processing controlsection 135 further determines whether or not the present angle ψ_(P) isequal to or less than threshold value ψ₂ (ST3207). If ψ_(P)≦ψ₂ (YES),the processing control section 135 determines that the above-describedstate S₁ is set and sets the region selection mode MS to the orthogonalcoordinate mode MS1 in the front hemispherical field of view (ST3208).

If ψ_(P)>ψ₂ (NO) at the above step of ST3207, the processing controlsection 135 determines that the above-described state S₂ is set and setsthe region selection mode MS to the polar coordinate mode MS2 in thelower hemispherical field of view (ST3212).

After setting the region selection mode MS to the polar coordinate modeMS2 at ST3204, the processing control section 135 reads the presentangle ψ_(P) again from the gyro-sensor 41 (ST3205) and determineswhether or not the present angle ψ_(P) is equal to or more than thethreshold value φ₃ (ST3206). If ψ_(P)≧ψ₃ (YES), the processing controlsection 135 determines that the image pickup section 11 has been changedto the above-described state S₁ and sets the region selection mode MS tothe orthogonal coordinate mode MS1 (ST3208). If ψ_(P)<ψ₃ (NO), theprocessing control section 135 maintains the state of the polarcoordinate mode MS2 (ST3204).

After setting the region selection mode MS to the orthogonal coordinatemode MS1 at ST3208, the processing control section 135 reads the presentangle ψ_(P) again from the gyro-sensor 41 (ST3209) and determineswhether or not the present angle ψ_(P) is equal to or less than thethreshold value ψ₄ (ST3210). If ω_(P)≧ψ₄ (YES), the processing controlsection 135 determines that the image pickup section 11 has been changedto the above-described state S₀ and sets the region selection mode MS tothe polar coordinate mode MS2 (ST3204).

If ψ_(P)>ψ₄ (NO) at ST3210, the processing control section 135 furtherdetermines whether or not the above-described present angle ψ_(P) isequal to or more than the threshold value ψ₅ (ST3211). If ψ_(P)≧ψ₅(YES), the processing control section 135 determines that the imagepickup section 11 has been changed to the above-described state S₂ andsets the region selection mode MS to the polar coordinate mode MS2(ST3212). If ψ_(P)<ψ₅ (NO), the processing control section 135 maintainsthe state of the orthogonal coordinate mode MS1 (ST3208).

After setting the region selection mode MS to the polar coordinate modeMS2 at ST3212, the processing control section 135 reads the presentangle ψ_(P) again from the gyro-sensor 41 (ST3213) and determineswhether or not the present angle ψ_(P) is equal to or less than thethreshold value ψ₆ (ST3214). If ψ_(P)<ψ₆ (NO), the processing controlsection 135 determines that the image pickup section 11 has been changedto the above-described state S₁ and sets the region selection mode MS tothe orthogonal coordinate mode MS1 (ST3208).

If ψ_(P)≧ψ₆ (YES) at ST3214, the processing control section 135maintains the state of the polar coordinate mode MS2 (ST3212).

In such a manner, by repeating the above processing, the processingcontrol section 135 automatically switches the region selection mode MSin accordance with the installation angle of the image pickup section11.

Herein, the following will describe a method for calculating coordinatesin a case of setting to the polar coordinate mode MS2 in states S₀ andS₂ or to the orthogonal coordinate mode MS1 in state S₁, and displayinga selected region display image Gsc in each of the modes. FIGS. 36, 37,38, and 39 are drawings concerning this coordinate calculation method.

First, as shown in FIG. 36(A), in the above-described polar coordinatemode MS2 (upper hemispherical field of view), it is assumed that theangle of Pan (rotation around a z-axis) and the angle of Tilt (rotationaround an x-axis or a y-axis) of a display plane 81 are H (x-axialdirection is 0 degree) and V (x-axial direction is o degree),respectively.

Next, in the orthogonal coordinate mode MS1, the direction of the fieldof view tilts by 90 degrees against that in the polar coordinate modeMS2, so that coordinate axes shown in FIG. 36(A) are exchanged toconvert the above-described Pan value and Tilt value.

Specifically, as shown in FIG. 36(B), respective axes in FIG. 36(A),i.e., the x-axis, the y-axis, and the z-axis are respectively exchangedwith the y-axis, the z-axis and rotation angles of the x-axis and a Panangle (H) and a Tilt angle (V) in FIG. 36(A) are respectively a Panangle (h) and a Tilt angle (v). In this case, a direction vector [D]that indicates a set direction of a selected region, which is a regionthat corresponds to an image to be displayed on the display section 14,is obtained in a matrix shown in FIG. 37 by rotating an x-axial unitvector in each coordinate system. With this, a sin value and a cos valueof each of the above-described post-rotation Pan angle (h) and Tiltangle (v) in the orthogonal coordinate mode MS1 are obtained as follows:sin(v)=−cos(H)cos(V)cos(v)=(1−sin(v)²)^(1/2)sin(h)=cos(H)cos(V)/cos(v)cos(h)=−sin(V)/cos(v)

It should be noted that h=H and v=V in the polar coordinate mode MS2.

On the other hand, if an output of fixed pixels such as video graphicsarray (VGA) display is used as the display section 14, the coordinatesof a display plane [a] that match this output will be represented asfollows:

[a]=[a00=(0, 0), a01=(r, 0), . . . a10=(0, q), a11=(r, q), a12=(2r, q),amn=(nr, mq), . . . aMN=(Nr, Mq)]

Assuming this display plane [a] to be a three-dimensional plane [P0] inwhich point sequences perpendicular to the x-axis are parallel to they-axis, the z-axis and a center thereof passes on the x-axis, and anx-coordinate is R (R=fish-eye radius, for example) as shown in FIG.38(B), the above-described plane [a] is three-dimensioned, as shown inFIG. 38(C) as follows:[a]=[(x0,y0),(x1,y0), . . . (xN,yM)]→[A]=[(x0,y0,1),(x1,y0,1), . . .(xN,yM,1)]

This matrix [A] is multiplied by a matrix [K] shown in FIG. 38(C) toprovide a matrix [P0] (=[K] [A]).

Subsequently, by using such parameters as shown in FIG. 39A that can beset on the above-described operation input screen Gu of FIGS. 15(A) and15(B), this plane [P0] is expanded and moved onto a sphere as shown inFIG. 39(B). In this case, assuming this post-movement plane to be aplane [P2], a point on the plane [P2] corresponding to a point Pj on theplane [P0] is a point Pj2 (x2, y2, z2). Each point on this plane [P2]can be obtained by a calculation equation of FIG. 39(D) by using thematrices of [P0], [P2], [X], [Y], [Z], and [M] shown in FIG. 39(C).

That is, [P2]=[Z] [Y] [X] [M] [P0] in the orthogonal coordinate mode MS1and [P2]=[Z] [Y] [M] [P0] in the polar coordinate mode MS2. Based on thethus calculated plane [P2] coordinate values, coordinate values (x2, y2,x2) that corresponds to the point Pj2 is used.

Then, in FIG. 39(B), by performing processing similar to theabove-described distortion correction processing principle explainedusing FIG. 12 based on the point Pj2 (x2, y2, z2) on the above-describedplane [P2], a focal point Qj2 that corresponds to the point Pj2 (x2, y2,z2) can be obtained. Further, by performing the similar processing oneach point on the plane [P2], it is possible to obtain the position ofeach pixel on the image pickup element 112 that make up a selectedregion display image Gsc obtained after the distortion correctionprocessing is performed in the orthogonal coordinate mode MS1 and thepolar coordinate mode MS2 and then use the pixel data of the position ofpixel thus obtained, thereby enabling the selected region display imageGsc in which no distortion is generated to display.

According to the image processing system 40 of the present embodiment,through the above-described constitutions and operations, theinstallation angle of the image pickup section 11 can be detected withthe gyro-sensor 41 to appropriately switch the region selection mode MScorresponding to this installation angle, thereby improving the userconvenience.

Further, since the region selection mode MS is switched on a conditionwhere a hysteresis is held, even when fluctuations in the installationangle are generated near the above-described threshold values ψ₁ and ψ₂,the region selection mode MS is prevented from being frequently switchedbased on the fluctuations, thus avoiding disturbing the user.

It should be noted that the direction detection sensor may be any othersensor, for example, a gravity sensor in place of the above-describedgyro-sensor 41, to detect the installation angle of the image pickupsection 11.

Further, although the present embodiment has switched the regionselection mode MS in accordance with the installation angle of the imagepickup section 11, the region selection mode MS may be switched inaccordance with, for example, whether or not the image pickup section 11has come into contact with an object.

FIG. 40 is a conceptual illustration of a method for switching thedisplay mode in accordance with contact. For example, in a case wherethe image pickup section 11 moves through a piping 95 in the directionof an arrow T, the processing control section 135 is set to, forexample, the polar coordinate mode MS2 in the piping 95, thereby pickingup the surroundings of the piping 95. Further, if it comes into contactwith a wall surface 96 at an end of the piping 95, the processingcontrol section 135 may be switched to the orthogonal coordinate modeMS1 to pick up this wall surface. In this case, the image processingsystem may be equipped with a defection sensor for detecting contact andapply the result of the detection to the processing control section 135in the image processing section 13. The detection sensor may be amechanical sensor or an optical sensor.

Further, automatic switchover between the orthogonal coordinate mode MS1and the polar coordinate mode MS2 can be performed without using thedirection detection sensor or any other detection sensor. Next, a casewill be described below in which the orthogonal coordinate mode MS1 andthe polar coordinate mode MS2 is switched from each other without usinga sensor.

Herein, to facilitate the understanding of the automatic switchover, thefollowing will be described with the field of view of the image pickupsection 11 being assumed to be, for example, 270 degrees. FIG. 41 showsa case where the image pickup optical section 111 has obtained a fieldof view of 270 degrees by using an ultra wide-angle lens. Light madeincident upon the image pickup optical section 111 goes toward the imagepickup element 112, to form a wide-field image Gc with a field of viewof 270 degrees on a sensor surface of the image pickup element 112.

Further, as shown in FIG. 42, if the image pickup section 11 having afield of view of 270 degrees is placed so that a direction of an arrowOCt that is a center direction of the field of view may be 45 degreesupward with respect to the horizontal direction, such a situation isgiven in which a front hemispherical field of view 53 h and an upperhemispherical field of view 53 u are obtained.

FIG. 43 show an example where the image pickup section 11 having a fieldof view of 270 degrees is placed. For example, the image pickup section11 is mounted to the bow of a boat in such a manner that the centerdirection of the field of view may be 45 degrees upward with respect tothe horizontal direction. Further, seats FS are set behind the imagepickup section 11. In a case where the image pickup section 11 ismounted in such a manner, if a forward landscape is displayed on thedisplay section 14 in the orthogonal coordinate mode MS1 suited for thecase of the front hemispherical field of view, it is possible to easilyset a selected region to a landscape in a desired direction, therebydisplaying a distortion-free image of the landscape in the desireddirection on the display section 14. Further, if a passenger sitting onany of the rear seats FS is displayed on the display section 14 in thepolar coordinate mode MS2 suited for the case of the upper or lowerhemispherical field of view, it is possible to easily set a selectedregion to a passenger in a desired direction, thereby displaying adistortion-free image of the passenger in the desired direction on thedisplay section 14. Accordingly, the image processing section 13automatically switches the region selection mode MS in accordance with adirection in which a selected region is set, thereby enabling an imagein which a distortion due to the image pickup optical section 111 iscorrected to display on the display section 14.

Herein, in a case where a selected region is set by specifying thedirection of the selected region, the range of an angle indicative ofthe range of the selected region, or the like by the input informationPS etc., the direction of the selected region can be determined on thebasis of the input information etc. that specifies the selected region.Further, since the selected region and the image region ARs correspondto each other, the direction in which the selected region is set can bedetermined from the image position of the image region ARs on whichdistortion correction processing is performed.

FIG. 44 show a case of automatically switching the region selection modeMS in accordance with the direction in which a selected region is set.If the image pickup section 11 is placed as shown in FIG. 43, thewide-field image Gc having a field of view of 270 degrees becomes theone shown in FIG. 44. It should be noted that if the image pickupsection 11 is mounted in such a manner that a center of the wide-fieldimage Gc may be in an optical axial direction of the image pickupoptical section 111 and the center direction of the field of view may be45 degrees upward with respect to the horizontal direction, thehorizontal front side position becomes, for example, such a point Pf onthe image as to have a field of view of 90 degrees.

In FIG. 44(A), if a region AS1 is given so as to include a front sideimage and a selected region is set in such a direction that, forexample, the center position of an image region ARs (not shown) may beincluded in the region AS1, the orthogonal coordinate mode MS1 is set.Further, if a region AS2 is given so as to include a rear side image anda selected region is set in such a direction that, for example, thecenter position of the image region ARs may be included in the regionAS2, the polar coordinate mode MS2 is set. It should be noted that ifthe center position of the image region ARs is not included in theregion AS1 or AS2, the set region selection mode MS is held.

Further, as shown in FIG. 44(B), a region of the wide-field image Gc maybe subdivided in a matrix beforehand to assign the region selection modeMS to be set to each of the sub-divided regions, then set the regionselection mode MS in accordance with any one of the regions in which,for example, the center position of the image region ARs is included.For example, to each region ASm1 including a front image, the orthogonalcoordinate mode MS1 is assigned beforehand as the region selection modeMS to be set. Further, to each region ASm2 including a rear image, thepolar coordinate mode MS2 is assigned beforehand as the region selectionmode MS to be set. Herein, if the center position of the image regionARs corresponding to a selected region corresponds to the region ASm1,the orthogonal coordinate mode MS1 is set. On the other hand, if thecenter position of the image region ARs corresponds to the region ASm2,the polar coordinate mode MS2 is set.

In such a manner, it is possible to automatically set the regionselection mode MS to an optimal mode in accordance with an image of anyfield of view displayed on the display section 14, thereby enabling theimage of a subject placed in a desired direction to display on thedisplay section 14 easily.

Further, since no sensor is used, it can be applied to any one of theimage processing systems 10, 20, and 30. It can also be applied to theimage processing system 40 using a sensor. In this case, byautomatically adjusting the positions or the region sizes of the regionsAS1, AS2, ASm1, and ASm2 corresponding to the inclination of the imagepickup section 11 based on the sensor signal ES from the gyro-sensor 41,even if the image pickup section 11 is not mounted so as to be 45degrees upward with respect to the horizontal direction, it is possibleto perform switch processing on the region selection mode MS with thesame characteristics as those in a case where the image pickup section11 is mounted so as to be 45 degrees upward with respect to thehorizontal direction. Further, if the positions or the region sizes ofthe regions AS1, AS2, ASm1, and ASm2 corresponding to the inclination ofthe image pickup section 11 can be set, it is also possible toarbitrarily set the switchover characteristics of the region selectionmode MS.

It should be noted that in the case of switching the region selectionmode MS automatically, by providing GUI display corresponding to each ofthe region selection modes, it is possible to easily decide which ofmodes the region selection mode MS is set to. FIG. 45 show GUI displaysand directions in which an image region ARs moves in the case ofswitching the region selection mode MS automatically. If the orthogonalcoordinate mode MS1 is set, as shown in FIG. 45(A), for example, the“Up”, “Down”, “Right”, and “Left” buttons are given as the directionbutton Gua2. It should be noted that FIG. 45(B) illustrates a directionin which an image region ARs moves in the entire image Gcp when thedirection button Gua2 is operated. If the polar coordinate mode MS2 isset, as shown in FIG. 45(C), for example, the “Center”, “Outer”, “Rightrotation”, and “Left rotation” buttons are given as the direction buttonGud2. It should be noted that FIG. 45(D) illustrates a direction inwhich an image region ARs moves in the entire image Gcp when thedirection button Gud2 is operated.

In such a manner, by providing GUI display corresponding to the regionselection modes, the user can easily decide which of modes the regionselection mode is set to. Further, when displaying on the displaysection 14 the image of a subject placed in a desired direction, theuser can easily select the direction buttons.

Moreover, the image pickup section 11 may stop the automatic switchoveroperations depending on a tilt angle with respect to the vertical (orhorizontal) direction and then either the orthogonal coordinate mode MS1or the polar coordinate mode MS2 is set. As shown in FIG. 46, if anangle ψ is in a range of “337.5 degrees ≦ψ<22.5 degrees” or “157.5degrees ≦ψ202.5 degrees”, the polar coordinate mode MS2 is setirrespective of the position of the image region ARs. If the angle ψ isin a range of “67.5 degrees ≦ψ<112.5 degrees” or “247.5 degrees ≦ψ<292.5degrees”, the orthogonal coordinate mode MS1 is set irrespective of theposition of the image region ARs. If the angle ψ is in a range of “22.5degrees ≦ψ<67.5 degrees”, “112.5 degrees ≦ψ<157.5 degrees”, “202.5degrees ≦ψ<247.5 degrees”, or “292.5 degrees ≦ψ<337.5 degrees”, theorthogonal coordinate mode MS1 or the polar coordinate mode MS2 isautomatically set as a combined mode in accordance with the position ofthe image region ARs as described above.

FIG. 47 is a flowchart showing switchover operations among the regionselection modes including the combined mode. The processing controlsection 135 performs angle detection to detect a tilt angle ψ of theimage pickup section 11 (ST3301). Next, the processing control section135 determines whether or not the combined mode is set based on thedetected tilt angle ψ (ST3302). If the combined mode is not set (NO),the processing control section 135 detects which one of the orthogonalcoordinate mode MS1 or the polar coordinate mode MS2 the regionselection mode should be set on the basis of the result of detection ofthe angle ψ (ST3303). If the combined mode is set (YES at ST3302), theprocessing control section 135 detects which one of the orthogonalcoordinate mode MS1 or the polar coordinate mode MS2 the regionselection mode should be set on the basis of the position of a selectedregion (ST3304).

If having completed detection of the region selection mode at ST3303 orST3304, the processing control section 135 sets the region selectionmode to the coordinate mode in which it is detected (ST3305). Next, theprocessing control section 135 provides GUI display corresponding to theregion selection mode which is set at ST3305 and the process returns toST3301 (ST3306). By thus switching the region selection modes, it ispossible to provide an interface which can be used by the user easily.

FIG. 48 is a flowchart showing operations when the direction button isoperated. The processing control section 135 determines whether or notthe region selection mode MS is set to the orthogonal coordinate modeMS1 (ST3401).

Herein, if the orthogonal coordinate mode MS1 is set (YES), theprocessing control section 135 determines whether or not the directionbutton is operated, based on the input information PS (ST3402). Ifhaving determined that no direction button is operated (NO), the processreturns to ST3401 while if having determined that the direction buttonis operated (YES), it is determined whether or not the “Right” button isoperated (ST3403).

If having determined that the “Right” button is operated (YES), theprocessing control section 135 performs processing such that a selectedregion is switched to a rightward position and the process returns toST3401 (ST3404). If having determined that the “Right” button is notoperated (NO at ST3403), the processing control section 135 determineswhether or not the “Left” button is operated (ST3405).

If having determined that the “Left” button is operated (YES), theprocessing control section 135 performs processing such that theselected region is switched to a leftward position and the processreturns to ST3401 (ST3406). If having determined that the “Left” buttonis not operated (NO at ST3405), the processing control section 135determines whether or not the “Up” button is operated (ST3407).

If having determined that the “Up” button is operated (YES), theprocessing control section 135 performs processing such that theselected region is switched to an upward position and the processreturns to ST3401 (ST3408). If having determined that the “UP” button isnot operated (NO at ST3407), the processing control section 135determines that the “Down” button is operated and so performs processingsuch that the selected region is switched to a downward position and theprocess returns to ST3401 (ST3409).

If having determined that the polar coordinate mode MS2 is set (NO atST3401), the processing control section 135 determines whether or notthe direction button is operated, based on the input information PS(ST3410). If having determined that no direction button is operated(NO), the process returns to ST3401 and, if having determined that thedirection button is operated (YES), it is determined whether or not the“Right rotation” button is operated (ST3411).

If having determined that the “Right rotation” button is operated (YES),the processing control section 135 performs processing to rotate theselected region rightward and the process returns to ST3401 (ST3412). Ifhaving determined that the “Right rotation” button is not operated (NOat ST3411), the processing control section 135 determines whether or notthe “Left rotation” button is operated (ST3413).

If having determined that the “Left rotation” button is operated (YES),the processing control section 135 performs processing to rotate theselected region leftward and the process returns to ST3401 (ST3414). Ifhaving determined that the “Left rotation” button is not operated (NO atST3413), the processing control section 135 determines whether or notthe “Center” button is operated (ST3415).

If having determined that the “Center” button is operated (YES), theprocessing control section 135 performs processing such that theselected region is switched to a central position and the processreturns to ST3401 (ST3416). If having determined that the “Center”button is not operated (NO at ST3415), the processing control section135 determines that the “Outer” button is operated and so performsprocessing such that the selected region is switched to a outwardposition that is a direction opposite to the central direction and theprocess returns to ST3401 (ST3417).

Through such processing, it is possible to switch the selected region toa desired direction easily. That is, it is possible to display the imageof a subject placed in a desired direction on the screen of the displaysection 14 in a condition where distortion due to the image pickupoptical section 111 is corrected.

However, the image processing system of the present embodiment has theabove-described plurality of modes as the display mode MH, so that if,for example, the selected image display mode MH2 is set, it isimpossible to decide which of direction of a field of view a selectedregion is set. Therefore, to confirm how to set a selected region, it isnecessary to perform any operations to change the display mode MH to theentire image display mode MH1 or the both display mode MH3. Accordingly,if a selected region is switched or the region selection mode MS isswitched, the display mode is changed so that an entire image Gcp may bedisplayed at least for a predetermined lapse of time, thereby making iteasy for the user to confirm the selected region even without changingthe display mode. It should be noted that a constitution of the imageprocessing system in the present embodiment is the same as that of theabove-described image processing system 40 shown in FIG. 32 and so itsexplanation is omitted.

FIGS. 49 and 50 show the operations of displaying an entire image Gcp,out of which FIG. 49 show a manner to change the display mode MH inresponse to the switchover of a selected region and FIG. 50 show amanner to change the display mode MH in response to the switchover ofthe region selection mode MS.

If the display mode is switched to a display mode in which the entireimage Gcp is not displayed or if it is instructed to switch a selectedregion or switch the region selection mode MS in a condition where adisplay mode in which the entire image Gcp is not displayed is set, theimage processing section 13 in the image processing system automaticallyalters to the display mode in which the entire image Gcp is displayedfor the predetermined lapse of time irrespective of the display modewhich is set.

FIG. 49(A) shows a case where an image is displayed in the selectedimage display mode MH2. If the selected region is switched in responseto a selected region switchover instruction, processing to combine theentire image Gcp with an image displayed corresponding to the setdisplay mode is performed and a post-combination image is displayed as anew display image. For example, the display mode MH is changed to theboth display mode MH3 to display the entire image Gcp also as shown inFIG. 49(B). Then, after the predetermined lapse of time elapses, thecombination processing of the entire image Gcp ends and the entire imageGcp is erased from the display screen. For example, the display mode MHreturns to the selected image display mode MH2 and the entire image Gcpis erased as shown in FIG. 49(C). Further, if the selected regionreturns to its original position in response to a selected regionswitchover instruction, the display mode MH alters to the both displaymode MH3 for the predetermined lapse of time and displays are performedas shown in FIGS. 49(C), 49(D), and 49(A) in this order.

FIG. 50(A) shows a case where an image is displayed in the selectedimage display mode MH2 and the region selection mode is set to the polarcoordinate mode MS2. Herein, if the region selection mode Ms is switchedto the orthogonal coordinate mode MS1, processing to combines the entireimage Gcp with an image displayed corresponding to the set display modeand a post-combination image is displayed as a new display image. Forexample, the display mode MH is changed to the both display mode MH3 andthe entire image Gcp is also displayed as shown in FIG. 50(B). Then,after the predetermined lapse of time elapses, the combinationprocessing of the entire image Gcp ends and the entire image Gcp iserased from the display screen. For example, the display mode MH returnsto the selected image display mode MH2 and the entire image Gcp iserased as shown in FIG. 50C.

Further, also if the region selection mode MS is switched from theorthogonal coordinate mode MS1 to the polar coordinate mode MS2, thedisplay mode MH alters to the both display mode MH3 for thepredetermined lapse of time and displays are performed as shown in FIGS.50(C), 50(D), and 50(A) in this order. It should be noted that thepredetermined lapse of time may be, but not limited to three seconds,five seconds, etc., for example.

Furthermore, the image processing section 13 can display the entireimage Gcp not only when the region selection mode MS is switched butalso when it is instructed to switch a selected region and, after thepredetermined lapse of time elapses, change it to such a mode in whichonly a selected region display image Gsc is displayed.

Further, the image processing section 13 may change the display mode forthe predetermined period of time not only when the selected regionswitching is generated in response to a selected region switchoverinstruction or the region selection mode MS is switched but also whenthe image of the image region ARs corresponding to the selected regionis changed in response to a change in a picked-up direction. Forexample, if the picked-up direction by the image pickup section 11changes, the image of a wide-field image Gc changes, so that theselected region display image Gsc also changes even if the position ofthe image region ARs on the sensor surface of the image pickup element112 is not changed. That is, a state such that the selected region ischanged occurs. Accordingly, for example, if having determined that animage pickup region is changed on the basis of the sensor signal ES fromthe gyro-sensor 41, the display mode alters for a predetermined periodof time to display the entire image Gcp. It is thus possible for a userto easily decide how it should be instructed to switch the selectedregion in order to display on the display section 14 the image of asubject placed in a desired direction.

FIG. 51 is a flowchart showing the flow of operations of the imageprocessing system in the case of changing the display mode shown in FIG.49 or 50. The processing control section 135 in the image processingsection 13 determines whether or not a selected region is switched(moved) (ST3601). That is, it is determined whether or not the inputinformation PS that indicates a user's operation on the above-describeddirection button Gua2 or Gud2 or the bearing button Gub2 for the purposeof selected region switchover instruction is supplied from the inputsection 12 or whether or not selected region switching is generated whena selected region is switched automatically on the basis of theabove-described trajectory information. Herein, if the selected regionis switched (YES), display control information JH is supplied to theimage output processing section 134 so that the image output processingsection 134 may display a combined image of the entire image Gcp and theselected region display image Gsc on the display section 14 (ST3606).Then, the processing control section 135 determines whether or not apredetermined lapse of time has elapsed since the output of thiscombined image (ST3607) and, if the predetermined lapse of time haselapsed (YES), supplies the display control information JH to the imageoutput processing section 134, to change the display from the combinedimage to the selected region display image Gsc from which the entireimage Gcp is erased (ST3608). That is, in a condition where the imageoutput processing section 134 is controlled by the processing controlsection 135, as shown in FIGS. 49 and 50, after the predetermined lapseof time has elapsed, the display mode MH is changed from the bothdisplay mode MH3 to the selected image display mode MH2.

If having determined that the selected region is not switched (NO) inthe determination whether or not the selected region is switched by theprocessing control section 135 in the image processing section 13, theprocessing control section 135 determines whether or not theabove-described display mode MH or region selection mode MS is switched,that is, whether or not the input information PS that indicates a user'soperation on the above-described “Select” button Gua1 or Gub1 or the“Menu” button Guc3 for the purpose of mode switchover is supplied fromthe input section 12 (ST3602). Herein, if the mode is switched (YES),the above-described steps of ST3606 and the subsequent are performed. Onthe other hand, if having determined that the mode is not switched (NO),the image processing section 13 determines whether or not the posture(installation angle) of the image pickup section 11 is changed (ST3603).That is, the processing control section 135 determines a differencebetween, for example, the result of detection by the gyro-sensor 41shown in FIG. 32 and a predetermined initial value based on the sensorsignal ES that indicates a tilt angle detected by the gyro-sensor 41. Ifthe installation angle of the image pickup section 11 is changed (YES),the image processing section 13 performs the processing of theabove-described steps of ST3606 and the subsequent.

Further, if having determined at ST3606 that the installation angle ofthe image pickup section 11 is not changed (NO), the processing controlsection 135 determines whether or not any user operation has beenperformed on the input section 12 and, if having determined that anyuser operation has been performed (YES), performs the processing of theabove-described steps of ST3606 and the subsequent and, if havingdetermined that no user operation has been performed (NO), repeats theprocessing of the above-described steps of ST3601 and the subsequent. Itshould be noted that if this user operation involves the switchover ofthe above-described display mode MH or region selection mode MS or theswitchover of a selected region, its processing will be the same as thatof the above-described step of ST3601 or ST3602, so that any otheruser's operations other than those are subject to be determined atST3604.

It should be noted that although in FIG. 51, if it is determined that aselected region is not switched, it has been determined whether or notthe mode is switched, whether or not the posture is changed, and whetheror not an operation is input in this order, it is possible to display anentire image Gcp for the predetermined lapse of time also bydetermination as to whether or not there is any of the selected regionswitchover, the mode switchover, the posture change, and an operationinput at ST3605 shown in FIG. 52 and, if any one of them is determinedto be positive, performing the processing of the steps of ST3606 and thesubsequent.

Through the above processing, if a selected region is switched, thedisplay mode MH or the region selection mode MS is switched, the postureof the image pickup section 11 is changed, or an operation is input fromthe user, such display is provided that the entire image Gcp may bedisplayed at least for a predetermined lapse of time. Therefore, if theselected region or the mode is switched, the user can easily confirm howthe selected region is set against the highlighted image Gs of the imageregion ARs in the entire image Gcp and also, after confirmation, observethe selected region display image Gsc without being blocked by theentire image Gcp.

It should be noted that, conversely an entire image Gcp has firstdisplayed in the both display mode MH3 and then, after a predeterminedlapse of time has elapsed, it is switched to the selected image displaymode MH2 in which the entire image Gcp is not displayed, the imageprocessing section 13 may first output only the selected region displayimage Gsc in the selected image display mode MH2 if the above-describeddisplay mode MH or region selection mode MS is switched, and, after thepredetermine lapse of time elapses, generate image data DVd so that acombined image may be displayed in the both display mode MH3 in whichthe entire image Gcp is combined. It is thus possible to confirm theselected region display image Gsc, with no blind areas, displayed on thedisplay section 14 and, after the predetermined lapse of time, observethe newly displayed selected region display image Gsc, and confirm thecurrently selected region.

FIG. 53 is an illustration showing another state of the display modeswitchover processing. When displaying the above-described combinedimage, the image processing section 13 may provide one-screen display asfor example, the above-described combined image in which the selectedregion display image Gsc is scaled down so that it may not overlap withthe entire image Gcp. Further, it may output the entire image Gcp bycombining it onto the selected region display image Gsc translucently.In either case, the image processing section 13 may perform suchprocessing that the user can observe the selected region display imageGsc with no blind areas in a condition where the entire image Gcp isdisplayed.

Furthermore, the image processing section 13 may change theabove-described predetermined lapse of time dynamically rather thanfixing it to a constant value. For example, if the region selection modeMS is switched, the position of a selected region may change greatly, sothat in such a case, the above-described predetermined lapse of time maybe made longer as compared with the above-described predetermined lapseof time in a case where there is any switchover of the selected regionin the same region selection mode MS or any change in the posture etc.of the image pickup section 11, so that the user can surely confirm theselected region. Further, for example, if the number of the sub-dividedregions increases in the divided display mode MH4, it may take long timeto confirm the plurality of selected regions, so that if the user hasperformed such an operation as to increase the number of sub-dividedregions (the number of highlighted images Gs in the entire image Gcp),the predetermined lapse of time may be prolonged as compared to that inthe case of switchover to any other mode or a change in posture so thatthe user can surely confirms the selected region.

INDUSTRIAL APPLICABILITY

In the present invention, in accordance with the switching of theselected region which indicates a part of a field of view represented byimage data, image data in which a subject image whose selected region ismade identifiable is not used in a display image is replaced by imagedata in which the subject image whose selected region is madeidentifiable is used in the display image. Therefore, it is well suitedfor a case where a desired region is set as a selected region in apicked up wide-field angle image to, for example, confirm the image ofthis selected region or the like.

1. An image processing apparatus that processes image data containingdistortion of an image pickup optical section, which is obtained bypicking up an optical image from a subject through the image pickupoptical section giving the distortion, the image processing apparatuscomprising: a data output section that outputs a distortion-correctedimage of a selected region; a display mode setting section that sets thedisplay mode; and a control section that, in accordance with switchingof the selected region, changes a first display mode in which thesubject image whose selected region is made identifiable is not used inthe display image to a second display mode in which the subject imagewhose selected region is made identifiable is used in the display image,and further wherein an image processing section that provides thedistortion corrected image is directly connected to both an input regionselection mechanism and a display section, wherein the control sectionhas a first region selection mode in which the selected region moves inan axial direction of an orthogonal coordinate system based on theswitchover instruction and a second region selection mode in which theselected region moves in a direction of an argument of a polarcoordinate system based on the switchover instruction and, in accordancewith switching of the selected region, changes the first display mode tothe second display mode.
 2. The image processing apparatus according toclaim 1, wherein in accordance with switching of the selected region,the control section changes the first display mode to the second displaymode for a predetermined period of time and changes it to the firstdisplay mode after the predetermined period of time elapses.
 3. Theimage processing apparatus according to claim 1, further comprising aninput section that performs switchover instruction of the selectedregion, wherein the control section switches the selected region inaccordance with the switchover instruction.
 4. An image processingapparatus that processes image data containing distortion of an imagepickup optical section, which is obtained by picking up an optical imagefrom a subject through the image pickup optical section giving thedistortion, the image processing apparatus comprising: a data outputsection that outputs a distortion-corrected image of a selected region;a display mode setting section that sets the display mode; and a controlsection that, in accordance with switching of the selected region,changes a first display mode in which the subject image whose selectedregion is made identifiable is not used in the display image to a seconddisplay mode in which the subject image whose selected region is madeidentifiable is used in the display image, and further wherein an imageprocessing section that provides the distortion corrected image isdirectly connected to both an input region selection mechanism and adisplay section, further comprising a direction detection sensor thatdetects an image pickup direction, wherein when having determined thatthe image pickup direction is changed based on a sensor signal from thedirection detection sensor, the control section changes the display modefor a predetermined period of time.